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AN OVERVIEW OF THE OREfiON STUDIES IN EDUCATIONAL RDD&E 



In the spring of 1970 the Training Branch of the U.S. 
Office of Education, National Center for Educational 
Research and Development, announced a plan to effect 
change in the preparation of educational RDD&E person- 
nel. Two factors led to the announcement. The underlying 
factor was the rather dramatic emergence in the past decade 
of development, diffusion, and evaluation activities as 
vehicles for educational improvement, and the attending 
need fur qualified personnel to carry them out. The 
precipitating factor, however, was evidence that in spite of 
an investment of approximately 30 million dollars by the 
Federal Government to help training programs become 
more responsive tc the personnel needs created by these 
new activities, essentially the same number and kind of 
personnel were being prepared in 1970 as in I96S. 

The plan for change reflected a strategy that can best be 
described as “beginning at the beginning.” It incorporated 
three interrelc.ted lines of activity: the creation of a 
conceptual and empirical base on which to build functional 
training programs; the design of more effective and efficient 
approaches to training; and the develupnwi.t of instruc- 
tional materials liiat reflect desired changes in both content 
and procedure. The propositions on which the plan rested 
were straightforward: ^ little was known about edu- 

cational development, diffusion and evaluation activities, or 
how they related to educational research; (b) even less was 
known about the training of personnel to carry out such 
activities; and (c) until both of these conditions were 
remedied the likelihood of designing effective and efficient 
programs to prepare personnel to carry them out was slight. 
The plan as a whole was coordinated so that the various 
activities within it would be developed with sensitivity to 
each other, and so that they would come together in 
completed fashion at approximately the same point in time. 
(For additional details on the plan (or change see Gupter I 
in Volume I of the series reporting the Oregon Studies.) 

The Oregon Studies, carried out by the Teaching 
Research Division of the Oregon Stale System of Higher 
Education, were to contribute in a beginning way to the 
conceptual and empirical base called for in the plan. As 
such they were to produce five products: a collection of 
detailed “case study" descrip'ions of projects that illus- 
trated exemplary RDD&E activities within various edu- 
cational contexts; a reliable, economically 'casible method- 
ology by which to collect the data needed to prepare the 
case studies; a conceptual system or framework for viewing 
the domain of educational RDD&E that could be used as a 
guide to the classes of data to be attended to in the case 
studies; cross-project analyses that highlighted the simi- 



larities and differences observed in the projects described, 
and that tested in rudimentary fashion the adequacy of the 
conceptual framework underlying those observations; and a 
compendium of the existing literature that pertained to 
either the nature of or the interactions between activities 
labeled educational research, development, diffusion and 
evaluation. These products are reported in five volumes: 



Volume I. Summary Report (with Technical 
Appendices) 

Volume II. The Literature of Educational RDD&E 
Part One (Research, Evaluation, and 
Development) 

Part Two (Diffusion & Combinations of 
RDD&E) 

Volume III. Conceptual Frameworks for Viewing 
Educational RDD&E 



Volume IV. Profiles of Exemplary Projects in 
Ivducational RDD&E 
Part One (Research and Evaluation) 
Part Two (Development) 

Part Three (Diffu^on) 

Volume V. A Methodology for the Study of 
Educational RDD&E 



Each volume in the series repotting the Studies has been 
designed to stand alone, but because each volume reports a 
different product, and each product can be understood 
fully only in relation to the other products, two “reader’s 
guides” to the series have been prepared. The first involves 
brief summaries or abstracts of the contents of each of the 
five volumes in the series. These appear on the inside of the 
back cover of the volume, and are intended to serve as a 
guide or overview to the series as a whole. A more detailed 
guide is provided by Volume I. In addition to serving as a 
genera] sumntary of the Studies, it contains descriptions of 
the developmental histories of the products reported in the 
various volumes, the reNtionships that exist between them, 
and the manner in which they have interacted over time. 
Accordingly, for the reader who wishes to determine 
quickly what each of the five volumes in tlie series contains, 
turn to the inside of the back cover of the volume; for the 
read**: who wishes to understand how the volumes relate to 
one another, follow.that by reading Volume I. 
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ABSTRACT 



This la ond of five vulumny reporting; results of the Oregon Studies 
In KclucaLlonal Research, Development, Diffusion, and Evaluation (RDD&E). 

It represents the output of a year-long effort to clarify and firm tlie 
conceptual bane that underlies educational RDD&E. It contains three 
commlBsioned papers, and a paper prepared by the Staff of the Oregon 
Studies. Dr«. Hendrik D. Gideonse, Gene V Glass, Blaine R. Worthen, 
and Leslie J. Briggs authored the commissioned papers; Dr. H. Del Schalock 
and Mr. G. Roger Sell authored the Oregon Studies paper. The volume also 
contains a formal critique of each paper, an author response to each 
critique, and an Introductory and summary critique of the papers as a set. 
Individually, the papers deal with one or more aspects of the domain of 
educational RDD&E. Collectively, they deal with the domain as a whole. 

With regard to area of emphasis, two papers cross-cut the domain, one 
focuses upon research and evaluation, and another upon development and 
diffusion. With regard to conceptual orientation, three papers are future- 
directed (prescriptive), and the one is present-directed (descriptive). 

With regard to functions examined, one deals with policy, two deal with 
operations, and one with management and operations. As a set, the con- 
ceptual papers and their critiques are intended to: (a^ serve as a bench- 

mark document with respect to the conceptual structures underlying educa- 
tional RDD&E; (^) serve as a primary reference for persons entering the 
field of educational RDD&E, and (£) serve as a stimulus to the continued 
dialogue that the papers in the volume have only begun. 
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The preBeiu volurue re[)re£ienL8 the results of a year-louR etfort Ij 
clarify and firm the conceptua] barte that underlies educational reseaTw-h, 
development f diffusion, am! evaluation (educational KhD^K). It rontaias 
three commissioned papers and a paper prepared by the Staff of t[ie Oregon 
Studies, each of whic[t was to provide an alternative view as to tlie nature 
and function of KDD\t wJtliln the context of educcition. It also contains a 
form;il critique of each j;aper, an author response to eacli critique, and 
an introductory and summary critique of the papers as a set. indlvidu.il iy, 
the papers deal with one or more aspects of the domain of educcUional Kidv.i:. 
Collectively, t[icy deal with the domain as a whole. 

Two purposes guided tlie preparation of the volume: (ja) it was to 

provide t!ie conceptual base for t[ie empir ical-»methodologlcal thrust within 
the Oregon Studies, and (h^) it was to serve a benchmark function with 
respect to the conceptual structure underlying the application of KODi^ik 
to the problems of education. In the role of conceptual guide to t[ie 
Oregon Studies, the t[iree commissioned papers served essentially as input 
to and context for the development by Staff members of the conceptual frame- 
work that served the actual guidon function in the Studies. That frame- 
work appears as the fourth paper in the volume. 

In the role of a benchmark document, all four papers, and their 
critiques, play a critical part. By order of appearance in the volume, 
the first paper provides a review of and an alternative to the basic assump- 
tions that have governed the practice of educational KDD&E, the second and 
third papers provide reviews of and alternatives to the concepts t[iat have 
been used to describe the nature and function of educational RDD&E, and the 
fourth paper provides a set of concepts that have been developed for and 
tested in the empirical investigation of educational RDD&E. Within this 
context, it was intended ttiat each of the papers would serve a benchmark 
function in relation to the particular role it played within the volume, 
and that the papers as a set would serve a benchmark function fot: educational 
RDD&E as an emerging discipline. The criteria and procedures followed in 
the preparation of th^ volume are described in the NOTES ON THE DEVELOPMENT 
OF THE CONCEPTUAL PAPERS that follows immediately after the Table of Contents. 

The volume was designed to be used by a relatively wide variety of 
readers. The fact that it was to provide the conceptual underpinnings to 
the Oregon Studies, and lUat it was to serve a benchmark function with 
respect to the conceptual structure of a discipline, required that It be 
written for consumption by advanced level professionals. The fact, however, 
that the conceptual strjcture of a discipline is the stuff that gives a 
discipline meaning, and the corresponding fact that anyone planning to 
enter a discipline must internalize its conceptual structure in order to 
function professionally within it, suggested that the volume should also be 
written for consumption by students, the trainers of students, and any who 
have recently entered the field. Accordingly, the volume has been cast 
in the framework of a search for sets of conceptual structures that have 
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uLlliry Llian Llio8e presently avallnl)le, and an effort lias l^eon 
made Ln j)rovi(je the sources of help and Interpretation that a relatively 
uninitiated reader mi^^ht need to Join meaningfully In that search. Py 
providing hucIi an orientation, by providing critiques of the papers that 
reflect tlie dilferenccs in point of view that exist In tlie field, and by 
providing Introductory and summary critiques to the volume as a whole, 

It is lioped that readers of widely differing barkgrouiicis will be able not 
only to make tlielr way through the volu;nc, but will be inclined when 
tiavlng done so to view the process of contlmu.d conceptual clarification 
as one of Mie most crilical of all tasks to be engaged in as a professional. 

Upon coi.'p luting the volume some readers may he disappointed in that 
it dues not resolve all the conceptual Issues that pervade ;:he field, or 
tliat it does not deal in greater detail with the actual operations of 
educational KDD&K, or that it does not provide clear policy and manage- 
ment guidelines relative to the funding of such activities. Others would 
liave been disappointed liad the volume attempted to bring closure to such 
matters. Whatever one's preferences might be, the intent of the volume 
was to strike a middle ground: provide as much synthesis as possible, 
but treat tliat synthesis as only one of an endless series of syntheses 
that must occur if the discipline of educational KDDiiE is to make a 
significant difference in the practice of education. Put in another way, 
the Intent of the volume was to bring about as much agreement as possible 
as to the nature and function of educational RDD&E, highlight the issues 
tliat remain, and provide those who wish to tussle with them a good con- 
ceptual leg on which to itand. The worth of the volume should be Judged 
in relation to that intent. 

As anyone who has had experience in the preparation of a volume of 
this kind knows, its completion requires the work and cooperdtlon of a 
great many people. First and foremost, it requires the willingness of 
highly independent souls to work together on a variety of writing tasks. 

The manner in which the time and effort of so many were articulated in 
this regard, the manner in which writing schedules were met — in spite of 
severe illness and hospitalization in the case of a writer and critiquer — 
and the manner in which other periodic demands of the editors were accommo- 
dated, were little short of miraculous. My sincerest appreciation to 
those whose names appear on the page which follows. 

Appreciation is extended also to Drs. Harry Ammerman, Norman Boyan, 

John Hopkins, Sam Sieber, and James Watson, continuing consultants to the 
Oregon Studies, for their contributions to the substance and design of 
the volume j to Drs. Cora Beebe, John Egermeier and Paul Messier of the 
U,S. Office of Education for their unfailing support of and cotitributious 
to the Studies as a whole; to Dr. James H. Beaird, continuing contributor 
to the volume and Director of Teaching Research, host institution for the 
Oregon Studies; to Mrs. Bernett Samples, Mrs. Irene Zimmerman, and the 
late Mrs. Jerl Jenkins as project typists and secretaries; and to Mr. Darrell 
Clukey and Mr. Norman Crowhurst as copy editors. Without the contributions 
of these persons the volume would have taken much different form. 
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Finally, I would like to express my appreciation to my colleague, 
Roger Sell. Mr. Sell served not only as a co-author in the preparation 
of one of the papers that appears in the volume, but as general coordi- 
nator for the preparation of the volume as a whole. He served in that 
capacity from the inception of the volume to its printing, and in so 
doing broughtto it a character that reflects his own. Without the 
likes of a Roger, volumes of this kind would not find their way into 
print so frequently. 



H. Del Schalock 

Director of the Oregon Studies 
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NOTES ON THE DEVELOPMENT OF THE CONCEPTUAL PAPERS 






Five apecifications guided the development of the conceptual papers: 

1. Each was to define, differentiate, and relate educational 
RDD&E activities, and provide a rationale for the particu- 
lar point of view taken in doing so; 

2. In combination, the papers were to provide a set of alter- 
native views as to the nature and function of RDD&E 
activities within the context of education; 

3. In combination, the papers were to be inclusive of the 
various segments of and points of view within the domain 
of educational RDD&E; 

4. In combination, the papers were to provide a synthesis and 
an extension of earlier conceptual efforts; and 

5. At Idast one of the papers was to be data dependent, that is, 
it was to be fashioned against and consistent in final form 
with the framework that guided the empirical thrust of the 
Oregon Studies.^ 

To meet these specifications, authors were sought who were both well recog- 
nized and respected for their contributions to the field, and holders of 
differing points of view as to the nature of and interdependencies between 
the elements within it. In addition, the papers were prepared in such a way 
as to assure that they would meet the criteria against which they were to be 
developed. 

Dr. Hendrik D. Gideonse, for six years Director of the Division of 
Planning for the U.S. Office of Education's National Center for Research 
and Development, and now a professional staff member of the Subcommittee on 



^The initial plan for the conceptual papers called for their synthesis 
into what was termed a ’’preferred" framework. That framework was then to 
serve as the conceptual base to the empirical and methodological efforts 
within the Oregon Studies. The assumption underlying the plan was not only 
that a synthesis could be achieved, bu that it should be — both for the 
benefit of the Oregon Studies and the field ac a whole. When the first 
drafts of the papers were reviewed, however, it was apparent that the notion 
of synthesis would have to be abandoned. The views presented in the papers 
were so diverse, and were held with such conviction, that the probability 
of reaching consensus on a single framework — especially in time for it to 
be of value to other aspects of the Studies — was near zero. As a conse- 
quence, it was decided that only the paper to be prepared by the Oregon 
Studies staff was to be data dependent, and the writers of the other papers 
were to be free to develop their respective ideas without regard to the 
criterion of data dependency or the desire for synthesis. 
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Executive Reorganiz«it ion and Crovernment Research, United States Senate, 
and Adjunct Professor of Poiicy Science at State University of New York 
at Buffalo, prepared the paper that appears first in the volume. His 
long experience in the management of RDD&E activities at the federal 
level. Ills interest and participation in poiicy decisions affecting such 
activities, aiui his work as an historian of tlie field, qualifies him 
uniquely to address the topic from the point of view of an ’’informed 
bureaucrat.*’ Prom that fraiae of reference Dr. Gideonse is of the opinion 
that the conception of research and development traditionally held within 
education is counLerproduct ive, and hr makes that opinion known in his 
paper, ’’Research and Development for Ikiu('ation: A Market Model.” Reflect- 
ing as it does a quest io/iinv; of the basic assumptions that have governed 
the practice of clucat iouaj. R)D^E, his paper provides a good point of 
departure for thinking generally about the role of RDD&E activities 
within the <^onLext of education. 

The second and third papers in the volume deal more directly with 
the practice of and interrelationships between educational RDD&E. The 
second paper, "Educational Inquiry and the Practice of Education,” 
auvliored by Drs. Gene V Glass and Blaine R. Worthen of the Laboratory 
for Educational Research, University of Colorado, focuses heavily upon 
the nature of educational research and evaluation, and their relation- 
ships to the improvement of education. Both Dr. Glass and Dr. Worthen 
are well known for their concern about these matters, and for their 
active role in the work of the American Educational Research Association. 

It is not accidental, therefore, that their paper reflects the ’’classic” 
or ”AERA” position with respect to the nature and function of RDD&E 
activities within the context of education. 

Dr. Leslie J. Briggs, Professor of Education at Florida State Univer- 
sity, and long recognized for his pioneering work in educational systems 
development, is author of the third paper, ’’Development and Diffusion as 
Mechanisms for Educational Improvement.” In his paper. Professor Briggs 
addresses the dual problems of developing demonstrably effective educa- 
tional practices, and getting those practices into wide use in the schools. 
After reviewing the literature that pertains to these issues, he outlines 
a research-based model of development that he believes incorporates within 
it the elements needed to assure both sound development and widespread 
diffusion, l^ile there is some overlap of effort between the Rrigge and 
Glass-Worthen papers, ic is not so great as to appear redundant.^ 



_ 

Only three papers were commissioned initially for inclusion in the 
volume. Upon the first review of those papers, however, it was the con- 
sensus of all writers and reviewers that a paper should be added that 
dealt expressly with the concepts of educational development and diffusion. 
Dr. Briggs, who was serving as a reviewer of the papers at the time, 
accepted the invitation to prepare such a paper. 
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The paper prepared by Dr. Schalock and Mr. Sell of the Oregon Studies 
staff differs from the other papers in that it sets the staf;e for and pro- 
vides the concepts used in the empirical thrust of the Oregon Studies. As 
suchi it provides basic definitions of the concepts of educational RDD&E, 
differentiates rather caretully between them, and spells out their inter- 
dependencies in largely operational terms. It provides a review of the 
existing literature as a basis for the definitions offered, and provides 
a rationale for thinking of RDD&E activities as "problem-solving tools" 
within the context of education. In contrast to the other papers, it does 
not deal with what "ought" to be with respect to the role of such activities 
in education, or make recommendations as to policy with respect to the 
long-term support and/or implementation of such activities. The title of 
the Schalock-Sell paper, "A Framework for the Analysis and Empirical 
Investigation of Educational RDD&E," accurately reflects its contents. 

In addition to the careful selection of authors as a means of assur- 
ing quality in the volume as a whole, and differences represented in point 
of view, each paper was submitted to a formal critique by an independent 
reviewer. Also, the papers as a set were submitted to formal critiques by 
two independent reviewers. Dr. David L. Clark, Dean of the School of Education, 
Indiana University, prepared the critique of the Gldeonse paper; Dr. Paul D. 
Hood, Director of the Communications Program, Far West Laboratory for Educa- 
tional Research and Development, prepared the critique of the Glass-Worthen 
paper; Dr. Norman J. Boyan, Dean of the Graduate School of Education, 
University of California, Santa Barbara, prepared the critique of the Briggs 
paper; and Dr. Glen Heathers, Professor of Education, Learning Research and 
Development Center, University of Pittsburgh, prepared the critique of the 
Schalock-Sell paper. Dr. J. Thomas Hastings, Director of the Center for 
Instructional Research and Curriculum Evaluation, University of Illinois, 
and Dr. Francis S. Chase, Educational Consultant, Professor (and Dean) 
Emeritus, The University of Chicago, prepared, respectively, the introduc- 
tory and summary critiques of the papers as a set. By providing for such 
critiques, and an opportunity for authors to respond to them, it was hoped 
that the strengths and weaknesses of the papers-- Individually and collec- 
tively — would be explicated. It was also hoped that it would open a 
dialogue at the conceptual level that would continue, for without such 
int ^rchange there is little hope that educational RDD&E will ever achieve 
the vitality and sophistication that it must if it is to make an observable 
difference in the practice of education. 

Finally, the procedure followed in the preparation of each of the 
papers in the volume, and to some extent their critiques, was also designed 
to further their quality and sensitivity to differing points of view. The 
first step in the process was to submit draft copies of the three base 
papers to public review (a two-day conference held on July 26 and 27, 1970), 
some three months after the papers had been commissioned. With one 
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exception all of the critiquers of the papers, the four continuing coa-- 
sultants to the Oregon Studies, ^ u.S, Office of Education representatives,^ 
and selected staff from the Studies took part in the review. Copies of 
the papers were made available to participants prior to the conference. 

The critiquer assigned a particular paper led the review of that paper at 
the conference, 

A second conference was held in mid-October, at vhich time a second 
draft of the initially commissioned papers, and a first draft of the Briggs 
paper, were reviewed. As in the first conference, the standing consultants 
to the project, U.S, Office of Education representatives, and selected staff 
members from the Oregon Studies took part in the review. In place of the 
critiquers, however, were the dirercora ot 12 USOE funded projects to design 
new training programs for the preparation of RDD&E personnel, (For a dis- 
cussion of the relationship between the training design projects and the 
Oregon Studies, see Chapter 1 in Volume I of the series reporting the Oregon 
Studies.) The rationale for the substitution was that input to the papers 
at that point in time by persons who had been thinking about similar issues, 
but within an applied context, would be of greater benefit than a second 
round of review by persons who were to have an opportunity to formally 
critique the papers at a later time. Copies of the papers pre'sented at the 
conference were sent to the critiquers. 

The July and October conferences constituted the only formally planned 
sources of input to the preparation of the papers and critiques. Authors 
used other sources, of course, but what those were, and when and how they 
were used, was a matter of their own choosing. No formal mechanism was 
established to provide for face-to-face interaction between critiquers and 
writers, or between critiquers and critiquers, beyond the July conference. 
Also, no formal provision was made for the interaction of the authors of 
the papers with the critiquers or with the editors of the volume beyond the 
July and October conferences. As a consequence, the form and substance of' 
both the papers and critiques are very much a matter of the author’s own 
style, point of view, and interpretation of role. 



^The continuing consultants to the Studies were Dr. Harry L. Aiomerman, 
an industrial psychologist with the Human Resources Research Organization; 

Dr. Samuel D. Sieber, a social psychologist at the Bureau of Applied Social 
Research, Columbia University; and Dr, James Watson, a social anthropologist 
at the University of Washington. Dr, John E, Hopkins of Indiana University, 
while technically a consultant to the U.S, Office of Education, also func- 
tioned as a continuing consultant to the project throughout its history. 
Following the October conference. Dr* Ammerman joined the project staff. 

^The U.S. Office of Education representatives attending the two con- 
ferences were Drs. Cora Beebe, from the Planning and Evaluation Division, 

John Egenneier, head of the Research Training Branch, and Paul Messier, at 
that time Director of the Division of Higher Education within the National 
Center for Research and Development. Dr, Messier is now directing imple- 
mentation planning for the proposed National Foundation for Higher Education. 
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Introduction 



You have located the proper document if you are one of those con- 
cerned with slowness (or lack!) of change in our schooling efforts. 

Also, if you are among those who are dismayed at the apparent lack of 
influence of basic research- - or even applied research--on the develop- 
ment of new materials, approaches, or goals for our educational enter- 
pri; ', you should give more than cursory examination to parts of all four 
position papers which form the body of this collection. Perhaps you are 
among those who believe that too many "things" get foisted into schooling 
(in which case that is your color-word, not mine) without sufficient 
information about worth from evaluation studies. Then read and learn — or 
read and wax polemic! Finally, if your "respected others" include those 
who bewail that all of those well-tested and research-based developments 
never seem to get adopted by many schools (and the chances are that they 
are development peoplel), then read sections of at least two of the papers 
which emphasize possible solutions to the dissemination problem. In short, 
if you are concerned with any aspects of research, development, dissemina- 
tion, and evaluation — in any order — in education, you are obligated to 
peruse the four papers and set about constructing better positions. On 
the other hand, if you feel that you have no concern with RDD&E — but are 
involved in education today — you owe yourself an awakening. 

Not that these papers are the only — and certainly not the last — 
word on such subjects. They are, however, carefully written and docu- 
mented arguments (and, for the most part, their assumptions are showing) 
for improving the quality of education through planning the interrelation- 
ships and strategies of that segment called RDD&E. O^.her aspects of the 
total educational undertaking could have been the focus. They should 
be elsewhere. For examples, try increasing (or decreasing) teachers' 
salaries or certification — or focus on altering the quantity and quality 
of local community participation — or wrestle with the problems of com- 
bining the pedagogical and nonpedagogical disciplines with each other 
and with our great social-political thinkers to solve our educational 
dissatisfactions. Each of these is worthy of thought, and some creep 
faintly into these papers. It remains that RDD&E constitute an important 
portion of the system in need of study. 
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The Range of Plana f ind St i atep,ies 



For convenience in alluding to the separate documents, 1 shall 
hereafter refer to them by the last names of the authors. 

The first paper, by Gideonse, comes very near to standing tlie old 
linear model of R, D, D, and E — done In that order — on Its head. Or, 
perhaps it transplants the head to the current foot-end. Without ignoring 
"R" and "E," Gideonse leaves me with the impression that the two "D" mem- 
bers of the quadrlvlal system are definitely to be rewired. 

Glass and Worthen concentrate on the meanings, the strategies, 
and the expectations for educational Inquiry and related activities. 

Their emphasis is heavily upon the "R" and the "E," albeit they do not 
ignore some of the knowledge about and arguments concerning both "D'u." 

Briggs treats the two 'D's" as a central concern, even though he 
demonstrates his awareness that a new configuration In these areas will 
have to be taken Into consideration by both "R" and "E" arrangements and 
goals . 

The final paper, by Schalock and Sell, concentrates with passion 
(and with rationale) on the need for further empirical study of align- 
ment of all four elements somewhat as they are beginning to exist . At 
the same time they point out the requirement for continuing support of 
both "traditional" research and a newer enabling research on RDD&E 
functions, activities, and relations. 

The document as a whole contains such varied concerns as arrange- 
ments for a finer grid of regional effort (Gideonse) and the need for 
a new reward system for evaluation within the research community (Glass- 
Worthen). There are suggestions for moving all of dissemination, and 
perhaps quantities of development, to the private sector and leaving 
inquiry to the universities and related structureu (Glass-Worthen) . 

These strategies are played against plans for having the major non- 
private effort in consumer needs and thereby dissemination (Gideonse) and 
(Briggs). In some places the document deals heavily with the training 
and nurture of special school personnel (Gideonse) , while In other pages 
the training of R & E personnel is stressed as an Important requirement 
(Glass-Worthen). A number of different schema or diagrams work heurls- 
tically to send one in different directions. Discussions of supporting 
personnel-needs (Schalock-Sell) and accounts of different modes of 
funding both appear (Glass-Worthen), along with presentations of examples 
of Innovation- or change-systems (Briggs) as well as schema for analysis 
of operations at different levels (Schalock-Sell). Happily, however, 
the range of topics and ideas has been limited to things which do bear 
dlscernlbly on educational RDD&E. The authors do not, In my estimation, 
waste time on telling of their pet projects. Although there Is some 
overlap In bibliography, the total range of documents cited Is grand to 
behold. 

This diversity of topics (and viewpoints) makes the total effort 
much more heuristic than didactic, although some sections within each 
paper take the latter mode Indeed. Perhaps someone should be commis- 
sioned to cut-and-paste (or to index) the document to arrange a "hand- 
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book** of ideas and plans for the system. If they do, I shall look at 
the product. But I prefer to give the worth-points for the volume’s 
present organization — compelling one to listen to arguments for dif- 
ferent approaches through papers reasonably consistent within themselves. 



Similarities — Dissimilarities 



Other reviewer; have been requested to look at and critique each 
document. My task, as I understand it, is to critique (I prefer **comment 
on*') the assemblage of the papers. However, I must compare and contrast 
within to get the flavor of the whole. If one were critiquing a paper 
by a single author, he would look for consistency-inconsistency . 1 find 

myself looking for the analogue across four sets of authors. But the 
word should never be consistency nor its antipode. Obviously, style is a 
dimension which attracts attention. Perhaps Glass-Worthen is the most 
informal style, with Schalock-Sell being, not its opposite (none is), but 
further down the line toward staunchly structured. This figures. Schalock 
Sell deals with an empirical attack on **what now lives** and **what may live. 
You had better be carefully structured and formal for that task. Schalock- 
Sell contains more categorical listings, taxonomic outlines, and **steps-to- 
be-taken** than the others. Gideonse and Briggs are structured and formal 
in a different sense. Each is consciously building the model — although 
neither carries that aura of **having found it** that many PPBS writers 
seem to have. None engages in turgid prose, although some readers who 
dislike carefully protected statements may think each does at times. The 
four are more similar than different in style (and now I have made enemies 
of eachl). 

A strong similarity across the papers is the extent to which each 
piles definition upon definition. As should be expected from the subject- 
matter, they tend to be defining the same terms. Furthermore, when they 
do define the same terms, there is a marvelous agreement for the most 
part. Exceptions to this include the definitions of evaluation by Glass- 
Worthen and by Gideonse. The latter tends to equate the two, especially 
if Stuf f lebeam's CIPP is the evaluation model. Glass-Worthen looks more 
often at the dissimilarity between research and evaluation. They do 
this with the rationale that most people see the similarities, but that 
now is a time that pointing out the differences may help support evalua- 
tion as a separate area. The Briggs definition of evaluation is less 
explicit than the others. He was, after all, dealing more narrowly with 
development and dissemination. Schalock-Sell, as I read it, could agree 
with both Glass-Worthen and Gideonse. This bothers me, but then their 
model calls for a study of what is. I strongly suspect that **what is** 
agrees with both — and many others. 

The differentiation of basic and applied research is another area 
of high similarity, if not total agreement. But then each uses some 
form of the definition supplied by NSF a few years ago. None of them 
seemed to say that all of educational research is applied, which is an 
attestation heard in significant quarters. I was a bit surprised by 
Glass-Worthen on that count, since I thought that Glass said educational 
research applied in his 1968 Research Paper No. 22 (see Glass-Worthen 
bibliography) . 



Tlie four papers are similar in that they all see a need to get 
new things into the schools, and they feel that we are doing this less 
efficiently now than is possible. Also, from these papers 1 get the 
feeling that each would depend on some form of RDD&E (different fixes, 
different mixes) to do this. 

All four papers see the need to look to other domains for sugges- 
tions as to what works, e.g,, to military, to hard sciences, to industry. 
Glass-Worthen is lightest on this push; Briggs is by far the heaviest; 
Gideonse and Schalock-Sell follow Glass-Worthen in that order. All 
recognize that the educational context is a social-political one, but 
first Gideonse and second Briggs make more ado about it. No paper sug- 
gests that education can copy the most successful model from other do- 
mains; agriculture, military, industry. And they seem to agree for 
the most part on the reasons. 

Briggs and Gideonse are strongest in appearing to make the assump- 
tion that if personnel from many educational areas are brought together 
to discuss aims and need there will emerge agreement. Briggs claims it. 
Gideonse seems to assume it. Glass-Worthen and Schalock-Sell are less 
explicit, but I get the feeling that they believe aims and methods of 
instruction- learning are better left in the hands of experts. Here we 
have two author sets in agreement when we contrast explicit statements 
of the other two sets — not because it was a debated issue among the 
four sets of the six authors. 

Another dimension along which we have similarities or dissimilari- 
ties is that of time of application of the **model." From where 1 sit, 
Glass-Worthen could start yesterday whereas Gideonse is definitely a 
future-oriented model. He states, '^Models can either describe present 
conditions or they can describe ideal states as possible future condi- 
tions. Both kinds of modeling are useful, but it is important to be clear 
which is being attempted. In the case of this paper, the model is 
future oriented to a state of affairs conceived to be desirable.” 
(Gideonse, p.l4 of manuscript) On this dimension, Briggs can start 
forming a new consortium in any one or more locations, of different 
sizes, with somewhat different mixes as of tomorrow. The main thing 
which must be present is a mi:c which includes persons from each domain: 
teachers , administrators, community, developers , and researchers . 
Evaluators seem to be included in the last named class. The Schalock- 
Sell model has begun. They have been doing empirical studies of RDD&E 
activities, organization, operations, outputs, and inputs. 

In general, the similarities of the four papers outweigh the dis- 
similarities two-to-one. This is comforting. It either results from 
a cult — or a party line — or it results from bright minds coming to 
agreement on a complicated problem. I am biased by knowing the people. 

But the evidence says that they arrived at these similarities (in the 
words I used before) ”by carefully written and documented arguments.” 

But then, I am connected with a university; but then, I have engaged 
in a moderate way in several of the four aspects under consideration; 
but then, I am one person reading the four position papers. I am 
willing to critique, but you read also! 
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Conflicts 



It is possible to have similarities with conflicts. It is possible 
to have dissimilarities without or with conflicts. For example: Ford 

Motor Car has similarities with Pontiac Motor Car. Their LTD and Bonne- 
ville, particularly, have much in common. But there are conflicts. A 
buyer cannot believe equally in each — unless he is a coin-flipper I Can a 
’’buyer'* of RDD&E believe equally in each of the four papers? Perhaps the 
better question is, ”To what extent can a ’buyer’ believe equally in the 
plans set by each paper?” 

As implied earlier, the four papers tend to treat different aspects 
of the educational-change problem related to RDD&E. It is difficult to 
obtain real conflict if one group is trying to improve the parks and 
another is wanting to improve tlie traffic flow. . .except in the funding 
area. Noninfinite resources call for conflict in implementation among 
different proposals for action. The Congress would see conflicts in the 
position papers. 

One obvious conflict between Glass-Worthen and Gideonse is in connection 
with the question of who initiates improvement. Quotes will show it better 
than I could describe it. First Glass-Worthen; "For some time to come, 
education is more likely to be improved through ingenious invention inspired 
by basic research and hard-headed evaluation of the inventions than by the 
direct application of the fruits of basic educational inquiry’’ (manuscript, 
p.lOl). Then Gideonse: ’’Successful innovations derived through research 

and development are more likely to arise from careful consideration of 
market requirements or possibilities than from the theoretical or technical 
possibilities emergent from R&D by itself” (manuscript, p. 21). Maybe they 
are not so far apart if the researchers are watching what is wanted (needed) 
in the schools. But as things stand in the original papers, concerned persons 
outside the establishment (the Congress, groups of taxpayers, involved 
parents) would see a difference. 

Another conflict arises when Schalock-Sell says to study the system 
empirically and try to improve it — and Briggs says forget it. Briggs 
wants to start a new system which has the school at the initiation point. 

My view is that the Congress should fund each on a small-scale try-out. 

But that is something whi^,h the Office of Education has never been able 
to do. For every situation, they take the approach of doing a monolithic 
job of including the total population. Politically, all states, terri- 
tories, and ethnic groups must be included. Therefore, one cannot try 
out two or more models at the same time. In my view of our present posi- 
tion, we need to limit our samples to much smaller regions or types and 
take multiple approaches. I am fully aware of the political necessity of 
giving "new" things to every region. I say that if a certain state, say X, 
does not seem to have much of a proposal, close your eyes and fund them 
anyway. 



^Exceptions to this may be Follow-Through and Triple-T. Both had some 
V7elcomed variation in treatment, but even in this case, there was expected 
generaliz ability of outcomes. No way I The variables involved were never 
equivalent . 



One possible final conflict among the paperi3 in this document 
concerns the use of the private sector as opposed to Federal and State 
support. Glass-Worthen , as stated previously, would turn most develop- 
ment and dissemination to private industry and business. Gideonse 
would bring them into a very narrow range of activities. Briggs and 
Schalock-Sell are less explicit in the use of the private sector, but 
I feel that each would use it if the money were there. 



Disappoin tments 

I looked at some length for one of the sets of authors to take 
a different view than that which says, "Data (information) about programs 
and their outcomes make a difference in decision-making at the upper 
levels." No one of them took the position that top-echelon people 
(Federal, State, or local) make decisions on the basis of the question, 
"What will get me the furthest — or get my unit the most power?" I can not 
help but think — from experience — that most policy bodies or individuals 
pay little attention to data unless the data fit their views. Those of 
us in RDD&E do believe in looking at the data. Most of those in bureauc- 
racies are looking for more power — more control. This is not necessarily 
denegrative, though it may be anti-quality- Improvement in some cases. 
However, I find no reference to this behavior in the four papers. Surely 
they know of the existence of such behavior! Especially should this be 
true of Gideonse. He does refer to politics, but not in these terms. 

I was disappointed also in the lack of attention to the politics 
of the local school system. This complaint is related to the foregoing 
one — but there are differences. Why should a given superintendent enter 
into a combine of schools and research centers with a real desire to 
innovate ? Innovations, by definition, are risks. If the new trick suc- 
ceeds (and he is in a cooperative with other districts), he does no 
better than they do. His board says, "forget it." If it fails, he is 
in the same sinking boat with the others. That is no help with the local 
taxpayers. Each top administrator needs to enter into agreements for in- 
novations for which takes the risk and, in most school situations, he 
feels that he should not take many risks. 

In most large school districts, the administrative support staff 
(supervisors, directors of curriculum, principals — for instance) are 
locked into a system which by its very nature opposes the initiation of 
change by individuals. In part, because of this, the teaching staff are 
conditioned against making (or having much to do with) policy matters. 

If a teacher really wants to do something very different, he is better 
off doing it quietly. In Briggs we get some excellent plans for forming 
consortia which will include RD&E people along with administrators, 
teachers, and community persons. But I was a trifle disappointed that 
there seemed to be insufficient discussion of the Immense problems that 
can be created by that mix. Education has witnessed some of the problems 
in the last few years in connection with Head Start and Follow-Through 
programs, among others. 

Finally, and briefly, I felt that there was far too little reference 
to the plurality of American education: different souls have different 
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goals. The tone of Schalock-Sell| to my ear, was one of ”a great thing 
Is developed, so all will use.” Mind you, they do not say that. Glass- 
Worthen makes no reference to the mix of values in any one school com- 
munity or on the national scene. Briggs actually suggests that one will 
get great consensus on aims and needs within the smaller consortia, al- 
though room is left for differing goals across them. Gideonse^s market 
model comes very close to providing for plurality, but it does not ex- 
plicitly deal with the problem and its consequences, at least as I read 
it. For example, can a school system administrator afford to take a 
very strong stand on a single set of objectives or needs in a community 
which includes a vocal mix of priorities? Isn't he better off forming 
consensus of the vague generalities of "quality education” or "improved 
citizenship” and the like, all of which are less than fully useful to 
the developers? 



The Worth 



My opening paragraphs do express my notion of the worth of the whole 
document. I am very glad that the large task was not assigned to one 
author or even a consensus-finding group of them. Professionals and lay 
policymakers should read all four position papers. They do contain a 
storehouse of ideas worth debating, trying, and knocking against each 
other. None of them is written with the pedantry of the panacea-giver. 
You too, I hope, will have some disappointments. And I trust that at 
least some of you will have different ones from mine and from each other. 
But I predict that most readers will have an overall reaction that says 
they found something well worth reading. 
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SYNOPSIS 



Jlie (iidcontie paper provldea a review of and an alternative to the 
baaic aBHumption that have governed educational RDDoK. It is argued in 
the paper that RDD&E activities are context specific, and that when app- 
lied within tlie context of education they assume different forms and roles 
than they do, for example, within the contexts of medicine, agriculture, 
or induHtry. It is also argued that, when applied to education, RDD&E are 
as much social-political activities (because they always involve tne nuitter 
of values) as they are scientific, and because of that, the decision struc- 
tures within them must incorporate persons at the receiving as well as the 
delivering end. In Gldeonsc'o terms such a view of educational KUD&E re- 
(luires . turn (ing on its) head (the) basically linear and hierarchical 
notions about how research and development can Improve educational practice, 
(and turn instead to) a highly sophisticated, imaginative, and unyielding 
concern for the market being served or created by educational R & D, that 
Is, for the consumers, the clients, and the users of the outcomes •••• " 
(p.l4) . 

Taken as a whole, Gideonse's conception of a market model for educa- 
tional RDU&E represents an excellent point of departure for a volume of 
this kind. It not only challenges a host of assumptions that those working 
in the domain take for granted, but offers an alternative set In their place 
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Introduction 



The second successive national administration — a Republican one 
this time — has embraced the cause of educational research and develop- 
ment, The Johnson Administration revised the basic authorizing legis- 
lation in 1965 and called for the establishment of a network of educa- 
tional laboratories "large and significant comparable in their way 
to the large-scale laboratories of the Defense or Atomic Energy estab- 
lishments .equal in size and scope to the major tasks they seek to 
accomplish."! The form of the Nixon Administration’s embrace has been 
the submission of legislation to create a National Institute of Educa- 
tion "modeled shamelessly on the National Institutes of Health."^ The 
backing and the promise seems high. Why then are we in such a muddle? 

Much speculation and discussion — and even some careful study 
and analysis — have been devoted to the role of researcr and develop- 
ment in improving educational practice. Major policy explorations 
have been completed, most recently the policy review conducted under 
the sponsorship of the Organization for Economic Cooperation and De- 
velopment (OECD). 3 Commissions, study groups. White House-appointed 
panels, interagency review committees at the Federal level, and other 
groups have addressed smaller or larger segments of the problem.^ The 
result 3f all this examination as far as policy is concerned seems to 
be a peculiar immobilizing self-consciousness. 



1 Letter from President Lyndon B. Johnson to Secretary John W. Gardner, 
July 5 , 1966 . 

2 Daniel P. Moynihan, White House press conference, March 3, 1970. 

3 For a brief sunimary of the OECD review see my "OECD Policy Review of 
U.S. Educational R&D," Educational Researcher , April, 1970. The 
status study I prepared in connection with the review has been pub- 
lished by the Government Printing Office and is available under the 
title Educational Research and Development in the United States 

(OE 12049) from the Superintendent of Documents, GPO. 

4 For a report of this large number of studies, up-to-date through 
becembei , 1969, see Chapter X of Educational Research and Develop- 
ment in the United States . At this writing (March, 1971) two more re- 
port"; should be added. One was prepared by the President’s Commission 
on Instructional Technology and ^s'ued in August, 1969. The other was 
prepared by Roger Levien of the RAND Corporation under contract to 
United States Office of Education as ^#^lanning study for the proposed 
National Institute of Education. 
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Politically the field of educational research has been in a state 
of disarray. Under fire from the Congress, the Bureau of the Budget 
(now the Office of Management and Budget) , the Office of Science and 
Technology, staff offices of the Department of Health, Education, and 
Welfare, other Associate Commissioners in USOE, the Chief State School 
Officers, the White House, and a swarm of self-appointed critics in 
and out of government, USOE*s research programs have struggled, stut- 
tered, and somehow survived, but each time at a slightly heightened 
level of exhaustion. 

Now, barely four years after the last set was launched, new initi- 
atives have been proposed. USOE has been pressed, for example, to move 
the management of the program away from the development of institutions 
to carry out R&D and toward the identification of specific research and 
development objectives which will govern the management of the program. 
The new proposal for a National Institute of Education which has been 
forwarded by the President to the Congress envisages a complete, wholly 
unspecified, but much needed upgrading and reorganization of the manage- 
ment and administration of educational research and development at the 
Federal level. The companion proposal for a National Foundation for 
Higher Education, while it appears to be a clear overlap of the pro- 
posed new Institute's functions, deserves mention, too. Finally, a 
new program of experimental schools originally proposed by former 
Secretary Finch has just been launched. Its appropriation for the 
first year exceeds the sum used to launch the first 10 educational 
laboratories in 1966. 

All of this has the Congress, the Chief State School Officers, 
the various and sundry parts of the Federal establishment, and various 
constituencies attendant to educational research programs hyperexcited, 
skeptical, horrified, hopeful, confused, and groggy at one and the 
same time. It should hardly be surprising, therefore, that accepting 
the assignment to prepare this paper has taken on for me something of 
the character of "going to the mountain." 

What better time could be found to take the most common, widely 
distributed, and basically linear and hierarchical notions about how 
research and development can improve educational practice^ and ex- 
plicitly turn them on their head? I propose in this paper that what 
is most likely to advance the field of educational R&D is not further 
worry and concern about the current state of the art in educational 



5 The basis for this statement is to be found in conclusions drawn by 
the four OCED examiners during the course of their policy review of 
American educational R&D. The full report of the review is unfor- 
tunately not yet published though more than a year has elapsed since 
the completion of the policy review. 
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R&D or the nature and interrelationships of R&D functions and processes.^ 
Wliat is needed instead is for us to develop a highly sophisticated, 
iinaginative , and unyielding concern for the market being served or 
created by educational R&D , that is , for the cons umers , the clients , 
and the users of the outcomes of such research. 



A Few Words About Modeling 



Why develop models? What is their purpose? How can they be useful? 

A simple answer is that models help us understand; they give us a 
sense of order to the ''blooming, buzzing confusion,*' They help sort out 
functions, ideas, or activities. Tliey help clarify relationships among 
elements . 

But this simple answer is clearly not sufficient itself. It only 
raises the question why we want that understanding. And the answer, I 
thifik, relates to some kind of instrumental need, a desire to do or 
accomplish something. Thus, we model to understand ^ order that we 
can make better decisions, manage better, change, self-fulfill better 
prophecies, and so on. A critically important point: this means that 

models are constructed with an eye to some kind of purposive action. 

Because this is so, it seemed to me important to try and sketch out my 
goal for the educational system as far as educational research is 
concerned. 

As I developed this paper, therefore, I tried at the same time 
to create a vision of the educational system as it might look if it 
were functioning with strong scientific support. This exercise, begun 
originally as part of my responsibilities as Director of Planning for 
USOE's National Center for Educational Research and Development, enabled 
me to move backwards and forwards in a continuing means/ends analysis. 

The conception I emerged with at the end of the appointed time for 
preparing this model can be found at the end of the paper. It is by 
no means complete, but I include it because it does provide the reader 
with some idea of the purpose I came to have in mind as I developed 
this particular model. 

A second point respecting models is the usefulness of pointing 
out the distinction between conceptual, logical, or ideal models and 
descriptive or empirical models. Models can either describe present 
conditions or they can describe ideal states as possible future conditions. 
Both kinds of modeling are useful, but it is important to be clear which is 
being attempted. In the case of this paper, the model is future oriented 
to a state of affairs conceived to be desirable* 



6 This tremendous concern for R&D functions and processes seems not to 
be repeated in any other field and is, I suspect, an outcome of two 
related phenomena: the supreme and long-prevalent methodological 

consciousness of educational research and the basic insecurity about 
our pretensions to scientific status. 
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A useful caveat is Chat, being purposive, modeling itself is a 
contextually related activity. Consider, for example, a complex system 
like a “muscle car/* The position such a mechanical marvel holds in le 
conceptual systems of a highway patrolman (e.g., a potential law violator 
or perpetrator of serious accidents), a twenty-two-year-old male (e.g., 
an aid to amatory success), or an ecologist (e.g., a serious example of 
human excess in terms of the drain on human and natural resources and 
the pollutants added to the sky-sewer) suggests the importance of know- 
ing where the modeler stands and what his purposes are. 

The same point can be made regarding a research project or a con- 
ceptual model. To the university research administrator, the Senator 
exercising Congressional oversight, the Federal research manager, the 
scientist directing the project, and the graduate school dean who wor- 
ries about financial assistance for two or three fewer graduate students, 
the project or model occupies a considerably different frame of reference. 
Thus R&D policy for education may be considered from the point of view 
of research scholars, policy-makers or practitioners in education, or 
from the peculiar vantage point of the student of social change. The 
same “objective” element tends to occupy very different places in the 
scheme of things depending upon who is doing the scheming, what his 
purposes are, and what the operative reward or incentive structures are. 

Given these kinds of understandings about modeling, it is important 
to state that what follows is future oriented, instrumental, ideal rather 
than actual, and oriented more towara the contextual settings of policy- 
makers for educational research and development rather than perfornK rs 
of such research or the users of its results. 

Applying this view of modeling to the immediate task, I am asked, 

“Why model educational R6fD?“. My response is that I wish to understand 
it better. And, if I am ther asked, *Vhy?” I reply that I want to under- 
stand it better so that R&D management can be improved. And, if I am 
then asked why I want to improve research management, I am pushed to 
a still higher order goal. My response is that the ultimate interest 
is in improving the service of our educating institutions to learners 
of all kinds . 

Means-ends analyses of this t 3 TJe are important because the char- 
acteristics and requirements of ultimate ends tend to shape the means 
that are chosen. They establish important parameters of action by 
projecting their shadow backwards from thv. desired goal or purpose to 
the present real field of action. 

Hence it becomes important to know what the frame of reference of 
the modeler is, how he views the ultimate end toward which his model- 
building is aimed, and what his operating assumptions are about the 
field or problem on which he is working. 
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Basic Assumptions 



It is important to identify what assumptions have been made in 
order to be able to undertake modeling^ to know what the modeler takes 
as his givens. 

Certain assumptions or judgments have been made which affect the 
process of developing a conceptual model of the applicati^lin of research 
and development to education. In the present instance I have been able 
to identify three areas where basic operating assumptions need expli- 
cation, These areas are the political and social character of behavior- 
al and social science, the character of change in large social systems, 
and the political structure of education. 



Unique Characteris tics of 
Behavioral and Social Science 



Much has been said about behavioral and social science. It has 
been called ”soft" as opposed to *^ard." It is dreadfully undersup- 
ported vis a vis the entire science budget in the Nation. It is often- 
times controversial and tends to be subjected to much more careful scru- 
tiny than biomedical or natural sciences. 

All of these statements are true. The point I would make here, 
however, underlies all those mentioned above and I think serves in 
large measure to explain them. The key question is how, if at all, the 
behavioral and social sciences (of which educational R&D is a part, in 
fact, the largest part) differ from other branches of science. 

For more than six years I worked as an administrator and planner 
of educational R&D. Month after month, year after year, I found myself 
continually amazed that scientific inquiry in the field of education 
should be the subject of so much foot-dragging, criticism, and contro- 
versy regarding policies, procedures, and support. No one appeared 
to understand what was being done or, if they did, agree with it. Every- 
one wanted to study it interminably. And despite the study, it is amazing 
how little seems to have been learned if the criterion is altered be- 
havior on the part of policy-makers. Thus we have the spectacle of a 
Counselor to the President and his staff on at least two occasions 
after the decision had already been made to press for ^the National 
Institute of Education (and presumably the staff work completed justi- 
fying that choice) offering the following evaluation of the field of 
educational R&D: 



(£) The Coleman Report showed certain target groups 
in American society were not being served by 
the schools as evidenced by their nonattain- 
ment of equal dis tr Ibution curves of achieve- 
ment . 

(b) The report further showed that the reason for 

this was that there did not exist adequate dif- 
ferential arrangements for the allocation of in- 
strumental resources (e.g. , dollars , trained 
manpower, curricula, techniques, etc.) so that 
equally distributed achievement patterns could 
be produced by the schools for varying target 
groups . 

(O If educational research and development had been 
performing well, it would have produced those 
techniques and sys terns of differential resource 
allocation • 

(^) Since they do not exist, educational R&D has 
clearly failed . 

(£) Therefore, we need a whole new initiative in 
this area. 

On the basis of the Coleman Report ? After having listened to this 
analysis, about all one can wish is that the Counselor in question had 
read his own words on the misapplication of the social sciences in the 
cause of policy development at the Federal level! 

Financial support since the original burst of enthusiasm in 1965 
has been begrudging and, in terms of purchasing power, declining. The 
political scraps and antagonisms within the Federal executive establish- 
ment, between it, the Congress, and the research community, and between 
all three and the educational establishment have been wondrous to behold. 
The result of the controversy has been disappointment, delay, procedural 
aberration, systematic encroachment on established administrative 
authority, and executive timidity. 

Only recently have I been able to articulate why this should be 
so and how the fact that it is, is probably the single most important 
characteristic of mission-oriented behavioral and social science research. 

Let me phrase it in the following way. The natural, physical, and 
biomedical sciences work on variables or entities that are different 
from those worked on in behavioral/social science research. The es- 
sential difference is two-fold. In the behavioral and social sciences 
virtually all of the objects of research or variables under study 
possess (or act as if they believe they possess) free will; and, 
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second, they are inextricably imbedded in a value structure of some 
kind or other. 

Let me be very clear about what I am saying here. I am not saying 
that the outcomes of physical sciences, biomedical sciences, etc., 
do not relate to choice or values. Clearly they do. What I ^ saying 
is that the materials, units, and variables with which and on which 
they v;ork are not themselves self-conscious, possessed of free will, or 
value laden. For example, atoms do not choose nor do chemicals or 
glands. But learners, and parents, and society, and institutions do. 
Rats and mice possess no human values (though we may anthropomorphize 
them from Lime to time). Neither are human values involved in the im- 
mediate intricacies of a high-energy physics experiment. But learning 
itself a value; failure in its achievement in an experiment involving 
real children or adults ^ a value question; and, all matters involving 
education or welfare or social futures are inextricably bound up in 
questions of worth and propriety and preference.^ 

The understanding is crucial. If behavioral and social science 
research is in its practice, as well as its implications, value-laden 
and choice-rich^ then science as it is practiced and managed in support 
of education is as much a social and political activity as it is a 
scientific one . This is an extremely important, if complicating, 
organizing principle. Its implications are far-ieaching . It affects 
who does research and development and where it is done. It forces 
a reconsideration of the decision-making structures which should exist 
for a behavioral and social science research program not only at the 
policy level where the objectives and targets are identified, but also 
at the technical level and perhaps even within the R&D operations them- 
selves . 

A political and social conception of behavioral and social 
science renders more immediately meaningful the remark attributed to 
Bernard Berelson when asked what he had concluded, having completed 
a huge compendium of research in the social sciences. He replied 
that the social significance of a research study is inversely propor- 
tional to its scientific quality. 8 While apparently antiintellectual 
on the surface, Berelson's remark reminds us that it is people, not 
things, who are affected by the behavioral and social sciences. And 
if the people who are to use that knowledge cannot understand it, or 
if it is so esoteric as to have little credibility,^ then one can 
question in what sense it is "knowledge*' or whether it should be 
imbued with power to affect decisions of any kind. 



7 Another way of saying the same thing is that in the behavioral and 
social sciences men are inside the systems being studied rather 
than outside. While the physical sciences are beginning to en- 
counter some analogous problems as the act of measurement alters 
the phenomena under study, the implications for the behavioral 
and social sciences of perception, judgment, feeling, attitude, 
and value are clearly much more problematical. 

8 Renorted by Lewis Eigen in Reappraisal of the Educational Technology 
Indus try . Urban Research Corporation, 1969. 

9 Because it has either sharply limited the variables or related them 
to one another in complex mathem§.tjj:al ways. 



Change Processes in Social Systems 



A second basic assumption or operating principle focuses on the 
process of social change. The argument is easy to sketch out. The 
educational system is a social system. The purpose of science is to 
develop understandings or knowledge which hopefully can be applied in 
some fashion or another to achieve desired human ends. The application 
of science to human or social systems is rarely a process that can be 
accomplished purely by decisions taken at critical points or times in 
the administrative structure. 

Change in a social svstem is not so much logical as it is psycho- 
logical, social, or political. We speak, for example, of an "idea 
whose time has come," What this means is that somehow a widespread 
willingness has developed to accept and act on the basis of a new 
conception or knowledge of some kind. But the phrase also implies a 
bit of knowledge in its own right, namely, that in social domains the 
"whose time has come" part of the phrase is far more important than the 
"idea." In other words, the conditions which create a readiness in 
a social field to accept an idea from science are more important as 
far as adoption is concerned than the idea itself. 

Hence the twin phenomena we find in all social fields. On the 
one hand we see the nonadoption of strong ideas in the absence of 
readiness. On the other, we have faddism which is nothing more than 
readiness to adopt in the absence of knowledge, a readiness whid> is 
soon disappointed by the low power of the innovation. The point is 
simple. For the improvement of education to come from science, the 
conditions causing practitioners and policy-makers to attend to the 
ideas emerging from science must be established parallel to and as 
part of the support of science itself. 



The Political Structure of the Educational System 

My assumptions should also be clarified respecting the political 
structure for education. At present the educational system in the 
United States Is multijurisdictional in character. Primary initiative 
for policy rests by constitutional authority in the States. Most of 
the States have in turn delegated this responsibility to local school 
districts. Any purposive model of educational research and develop- 
ment needs to address head-on what its assumptions are about the 
present and future character of the major decision structures for the 
educational system as a whole. 

In this regard I want to make clear my assumptions that policy- 
making respecting instruction and curriculum will continue to be 
decentralized. I hold this view not because that is the way it is 
now and, being strong, it is likely to continue that way. Rather 
than seeing such a situation as anachronistic (as many people do these 
days though perhaps not openly admitting it), decentralization is 
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very much the direction that is being made possible and is in fact 
occurring in a variety of other areas in the contemporary world. 

The communications revolution and, cliche though it may be, the 
general elevation of the level of popular knowledge make possible 
and indeed create demands for decentralization that would have been 
thought inconceivable (if not undesirable) a decade or two ago. The 
telephone, the Xerox machine, and the mass media assure the availability 
of information to large numbers of people. The immediate presumption is 
that good communications strengthen the center by making it possible 
for everybody uo get the right word. \>/hnt actually happens, though, is 
that people who aie close to the action now not only know what they 
know, but they also know what the policy-maker at the center knows. In 
such circumstances the demand to make decisions at the periphery will 
be loud and insistent. This trend is likely to contittue at, I believe, 
an increased rate. 

These assumptions respecting the political basis for ^he institution- 
alization of education is nationally specific (meaning that it is linked 
to a particular culturally imbedded structure for education). My 
conclusion is that it dovetails with the eai^ier expressed view of the 
political and social character of behavior and social science. It is 
a different point, however, not an identical one. 

An observation here might be useful regarding the degree to which 
this assumption of decentralized policy development and execution and 
the assumption about the political and social character of behavioral 
and social sciences relate to one another, I recently attended a UNESCO- 
sponsored working party at which we discussed the application of systems 
analysis to the problem of innovation in education, with particular re- 
ference to educational technology . One of the key items on the agenda 
was the consideration of what the UNESCO Secretariat called conversion 
strategies or what is referred to in this country more often as dif- 
fusion or change process. That the question was as much political and 
philosophical as it was instrumental became clear as important differ- 
ences emerged between Cartesians and empiricists, between French- 
speaking and English-speaking, find between nationals from centralized 
systems and nationals from decentralized systems. The English-speaking, 
empiricist, decentralized system representatives seemed generally to 
worry much more about change process that the Cartesian, French-speaking, 
centralized system participants. 

Several intriguing questions arise from this observation. For 
example, to what extent are we observing an instance of the Whorfian 
hypothesis that language carves out the reality we see? Does the 
logical-derivative character of French as a language affect their 
perceptions of change problems in ways different from the empirical- 
generalization character of English? Does the language itself only 
express the differences implicit in basic philosophical distinctions 
between empiricists and rationalists which in turn tend to produce 
decentralized vs, centralized educational systems? And most intri- 
guing of all, does what one believes affect in an objective way the 
kind of problem which exists? For example, the Americans and English 



tended to give short-shrift to the notion of the printed word as an 
effective conversion strategy. But the Frencn and Belgian represen- 
tatives insisted that for them publications were not only effective 
but essential, especially if the publications came from the center. 

This aside illustrates some of the dimensions of the problem 
with which we wrestle as we build models, and in particular, the 
possibility that in the field of behavioral and social science 
policy we may not be building universal models but rather national 
or culturally-specif ic ones. 



A Market Model of Educational 
Research and Development 



From the basic assumptions and other considerations developed above 
— the purposes and contextual relatedness of modeling, the political 
and social character of the behavioral and social sciences, and the 
socio-psychological dimensions of change in social systems — I am led 
to one compelling conviction. Educational research and developmeut must 
be conceived in terms of the market, consumers, and clients it is ^supposed 
to serve. Only af t er that principle is firmly established should atte n- 
tion be directed to the processes, techniques, and functions which might 
accomplish that servic e . 

Prerequisite to the application of science to education is the 
examination and redefinition of what the education market is, what it 
means to consider clients or practitioners as a ’’market,” and how to 
translate market requirements (conceived either in present terms or 
desired future terms) into product or outcome statements that will pro- 
vide useful guidance to the development and management of research and 
development policies and practices. Successful innovations derived 
through research and development are more likely to arise from careful 
consideration of market requirements or possibilities than from the 
theoretical and technical possibilities emergent from R&D by itself. 



10 These conclusions were formulated well before I encountered a mono- 
graph published by the National Science Foundation (NSF 69-17) , 
Successful Industrial I nnovations , by Sumner Myers and Donald G. 
Marquis. After stating their primary conclusion that technical 
change was to a significant extent based on the cumulative effect 
of small, incremental innovations, their second conclusion was that 
’’recognition of demand is a more frequent factor in innovation than 
recognition of technical potential.” The only caveat I would make 
respecting the idea of a market model is that I am not referring 
here to any notion of ’’free market economies” wherein that which 
sells best is ”best,” or somehow foreordained, or good. In short, 
sales are emphatically not the crl teria which determine success. 

The criterion referent which is most important, ultimately, is 
client satisfaction — are individuals and the society receiving 
from education what they desire? 




A Conceptual Turnabout 



Theodore Levitt, writing 10 years ago in the Harvard Business 
Review , analyzed the phenomenon of growth Indus tricsT^-*- He was inter- 
ested in exploring what a growth industry was and what kind of advice 
he might give regarding any given industry’s position with respect to 
such growth. Levitt concluded that there was no such thing as a growth 
industry per se, only far-sighted, successful management. He concluded 
industries were threatened, slowed, or stopped not because markets were 
saturated but because there had been a failure of management. 

Levitt presented brief analyses of railroads and the Holly\^ood film 



industry as classic examples of old ’’growth industries” gone sour. He 
described dry cleaning, electric utilities, and oil as industries in the 
shadow of obsolescence. He concluded that there was no such thing as 
a growth indus try , only "companies organized and operated to create and 
capitalize on growth opportunities.” Levitt (1960, pp . 47-48) identified 
four conditions which usually guaranteed the cycle of bountiful expansion 
and undetected decay : 




(.a) 


The belief that growth is assured hy an 
expanding and more affluent population. 






(b) 


The belief that there is no competitive 
substitute for the industry’s major 
product . 






(O 


Too much faith in mass production and in 
che advantages of rapidly declining units 
costs as output rises. 






(d) 


Preoccupation with a product that lends 
itself to carefully controlled scientific 
experimentation , improvement , and manu- 
facturing cost reduction. 





Levitt’s article then analyzed these four conditions in some detail. 
The central conclusion he arrived at was that healthy industries see their 
primary role as customer-satisfying rather than goods producing. 

”An industry begins with the customer and his needs, 
not with a patent, a raw material, or a selling skill. 

Given the customer’s needs, the industry develops 
backwards, first concerning itself with the physical 



11 Theodore Levitt, ’’Marketing Myopia,” Harvard Business Review , July- 
Augus t , 1960 , pp . 45-46 . 



delivery of customer satisfactions. Ther it moves 
back further to creating the things by which tlioso 
satisfactions are In part achieved ... .Final ly , tlie 
industry moves back still further to f inding the 
raw materials necessary for making its products ” 

(p. 55). 

Of course there are some obvious problems in directly applying the 
form or the conclusions of any analysis of the production of goods or 
the health of a bus iness or Indus try to the social service doma i ii . But 
there are intriguing points of comparison which can be made. For exampl 
we might define the customer's needs as those skills requisite f<>r 
success in school and the delivery system choices as Inschool meclianisms 
or new institutions like Head Start , preschool television programming , 
or wlintever. Educational development would then entail constructing 
the programs, organizational models, instructional sequences, and staff 
capabilities. And, the raw materials would include the knowledge base 
in both its specialized forms ami as it exists widely diffused about 
us all . 

But the most important comparison of all is the distinctl 
between producing goods (or services) and satisfying customers. 

The key point for the purposes of this paper is that those industries 
that have maintained a posture of sniisfying customers have thrived; 
those that have concentrated on producing goods have either stabilized 
or gotten into serious difficulties. 

Applied to educationrl R&I) Levitt's conception leads to a funda- 
mental reorientation of the status hierarchies implicit in classical 
research- development-dissemination-adoption (R-D-D-A) models of ed- 
ucational research. It suggests, for example, that before research 
and development can be used ("bought" in the literal and figurative 
senses) it must be delivered; and, before it can be delivered it must 
in some sense be needed (even trough the need may have to be "created" 
rather than merely identified as preexisting). A market model sug- 
gests, therefore, that attention to the conditions Cor change precedes 
the search for and development of specific changes. It suggests that 
how we are to reform and improve education shapes how we define and 
use research and development to that end. 12 

The market model entails giving up as futile the idea that the 
educational system will change as a consequence of external levers of 
research and development. Rather than being the lever of change, re- 
search and development is Che fulcrum. Adoption of a market model for 
conceptualizing and managing educational research and development, how- 
ever, should be done with the understanding that it is largely content- 



12 This, of course, is the reverse of the operating assumption of OECD 
examiners encountered in their discussions acr iss the United States 
with researchers, developers, and policy-makers of different kinds. 
They found the most prevalent view was that how research and develop- 
ment was pursued would determine hciw the ayatem would (diarige and In 
what ways. In nhort , (lie examiners dlHcoverod ii mudi strongur 
and conrorri for the product and ita pmtumXiiI tfifeacy Llian lor (lu* 
market or customers being aerved, howi^ver tCer might be Iclcnti' 
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less with respect to specific models of the Interrelationship of various 
research, development, or research-related functions. In that sense, the 
iiuirket model defines the game, not the game plan. Linear models might 
fit. Linkage models might fit. Output models, process models, or feed- 
back models might -'Iso flt.^^ While later on T propose ways of cor- 
ceiving of research and development which seem most generally rompaciblc 
with the market notion, the point is that the examination of the conditions, 
reasons, and requirements for change within the educational system itself 
is the first necessary step in the improvement process. 

Anotl er implication of the market model is tliat the primary referent 
of concern should be the client (conceived of as either learner or ed- 
ucationaJ practitioner — more on this later) rather than scientific 
theory. This is similar to Levitt's idea but casts it specifically in 
terms that are more commonly associated with educational research. By 
asserting the prime importance of the client and the practitioner, 
it makes all the participants — theorists, scientists, engineers, 
polic/ analysts, teachers, administrators, and the learners themselves — 
equals in a transaction (after Eric Berne) process. The central feature 
of that process is co-equal presentation of needs, knowledge, problems, 
and possible solutions. The status hleratchles implicit in some 
models of research and development dissolve in the market model and are 
replaced by transactional processes of confrontation, bargaining, or 
negotiation among equals. 



13 Sec Chapter I of Educatlona] Research and Development In the United 
States for discussion of the terms used here* 

14 This conception seems entirely compatible with ideas developed by 
Sam D. Sieber in his paper "Images of the Practitioner and Strategies 
for Inducing Educational Cnange," Bureau of Applied Social Research, 
Columbia University, February, 1967. Sieber presents four strategies 
for change based on the images of the practitioner as rational man, 
cooperator, powerless participant, and status occupant. He analyzes 
each in terms of the locus of change, the channels of influence, the 
change agent, and the efficient'' of the change measured in terms of 
effort, coverage, and yield. Sieber's preferred status occupant 
strategy based on the writing of Merton and Gross, Mason, and 
McEachern, while using quite different language and concepts, comes 
very close to expressing the same kind of transactional concept 
identified above . 
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How is ’^arket*^ To Be Defined? 



One of the most powerful features of the market model is that it 
forces consideration of who the market really is when discussing 
educational research and dev< lopment, For starters, what seems quite clear 
is that the market is not primarily researchers and developers operating 
in tie field of education. Wliile saying this may be obvious, much of the 
rhetoric spun out over the past 10 years or so about educational research 
and much of the actual decision-making respecting educational research 
has had the operational effect of attending more to the needs of the 
R&D community as a market than to schools, practitioners, parents, or 
children . 

The market model focuses attention in quite different directions. 

It lecids lo questions about which market, how we are to conceive of it, 
and what our time frame is. 

Is the system or the client the market? The first prime issue 
arising from the market model stems from considering who the market 
is: the system which carries out educational functions or the clients 

being served by that system? In short, whose needs (should) define 
the market? 

Careful attention reveals that as an economy, education, like all 
social services, must be considered in a rather different fashion. 

In most market economies with which we are familiar, the consumer, 
ultimate purchaser, or user are all the same person be he individual 
or corporate. For social services, however, the ultimate client or 
user is only rarely the direct purchaser. In fact, it would be far 
more accurate to say that the ultimate client for education, relative 
to the service being offered, stalls more often as recipient rather 
than consumer. 

In a sense, I am playing a word game here, but it is an inq)ortant 
one. For by the term ''recipient** I mean to imply a degree of passivity. 
This can be contrasted to the more active, capable-of-demanding-account- 
ability implication of the term "consumer." In fact, the buyers of 
behavioral and social technology for use in education are almost always 
intermediate to the ultimate beneficiary. The fact. that the market as 
purchaser is intermediate to the market as ultimate recipient of 
services is of crucial importance. The market model causes this question 
to be raised in head-on fashion. 

A second sense in which attention needs to be directed to the issue 
of who the market is and who the client is lies in the classic tension 
between individuals as clients and the society as client. Heightened 
focus on the market and how it is defined assures that this concern 
will not be glossed over. No claim is made \ere that only a market model 
will cause the individual/society issue to come more sharply into focus. 
Rather, it appears only that such issues are more likely to be addressed 
systematically and continuously as a consequence of the particular per- 
spective that the market model affords. 
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Actual or present markets vs. Ideal or future * A second key issue 
su»"faces almost immediately. Should existing needs as currently defined 
by client groups (in either sense as discussed above) constitute the 
market or should the market be conceived in terms of needs defined if 
conditions and understandings were as they should be? Does a market 
model require that customers be satisfied only immediately or in the 
long run, too? Does it mean that R&D has to provide products which 
meet felt ("low-level") needs or can it hope to meet real ("high-level") 
needs? 

I have deliberately used ambiguous and perhaps loaded language in 
these questions, I wish only to make clear that there is an issue here 
which needs to be dealt with. It can be perhaps most bluntly explicated 
by stating that there are any number of people in R&D and academia who 
believe that they know better what the needs are than clients or practi- 
tioners do. Whether the belief is justif .ed is a problem in its own 
right — a political problem, too, I mighc add, since it deals with 
fundamental questions of who has the right to define and choose means 
and ends in this domain. 

One way to deal with this kind of problem is through the systematic 
use of means/erds analyses and in rigorous comparisons of actual ac- 
complishments to those which are desired. Means /ends analyses can help 
resolve the market question by producing rigorous dialogue focusing on 
whether that which is desired will really have the effects or outcomes 
intended. Atuending to discrepancies between desired and actual achieve- 
ments of the educational services will similarly yield important input 
for resolving which needs statements of practitioners or clients have 
priority (presuming, always of course, that the statements of desired 
outcomes have legitimacy and standing). 

What is most important, however, is the pressure to hold a dialog 
between clients (or their representatives), practitioners, and per- 
formers of R&D to iron this question out in a manner satisfactory 
to all. We come back, in short, to the notion of a continuing 
transactional process among equals. 

Other terms for considering this question are more common to 
industrial research operations. One finds reference in such settings 
to concepts of "defensive" and "offensive" research. Defensive research 
is that undertaken to improve the efficiency, effectiveness, or cost of 
existing products, services, or functions. Offensive research is that 
conducted to develop wholly new product lines serving newly defined 
customer needs or which replace an existing, product by one which comes 
at the need in a wholly new way. 

Thus defensive research by a corporation would improve detergents; 
offensive research would work on synthetic fabrics which resisted or 
prevented soiling. Defensive research would improve gasoline for auto 
fuel; offensive research would allocate large suras to the development 



of acceptable electric or steam automobiles. Defensive research would 
improve teachers’ behavior in the classroom; offensive research would 
develop home television programming which attempts to guarantee that 
children will enter school with an established competence in reading, 
writing, speaking, and listening. 

Perhaps most important, however, in considering actual vs. ideal 
markets for education is the problem of delivery systems. Levitt 
asserts that after needs are considered, the first thing to be addressed, 
even before a product or service is defined, is the nature and adequacy 
of the systems for delivering customer satisfaction. This point is 
critical, especially for the educational system organized as it is. 
Decentralized, hierarchically structured in the bureaucratic sense, it 
is curiously ’’flat” if the object is to reach nearly three million 
teaching professionals. All sorts of innovations might conceivably be 
designed, tested, and validated; but, if there is no delivery system, 
then they caa be introduced only with the greatest of difficulty, if at 
all. 



It is with respect to this issue that some of the most difficult 
problems will surely arise. The existing system is not geared for 
renewal. Its structure does not provide easily for the continuing 
education of professionals; its incentive systems do not stimulate 
individual initiative as in other professional fields. 

Here is where the vision of an ideal system sketched in at the 
end of this paper may serve a dual purpose. Not only does it constitute 
an important part of the conception underlying the market model, but 
it comprises a beginning statement of a more ideal market as a deliv- 
ery system. It constitutes, therefore, a reference point in the future 
against which certain kinds of change-enabling R&D work might usefully 
be done. 



The Conceptualization of Research Functions 
Under the Marke t Model 



Conceiving of R&D functions in a market-oriented fashion helps 
get us out of at least one box in which we have struggled periodically. 
Over the past few years a number of analytical tools have been developed 
to help plan and manage research programs. At first it was thought 
that if we could develop clear, simplified terms grouped in such a way 
that all the terms in a given class were mutually exclusive, we would 
have powerful tools which could then be used to decide where funds 
should go for R&D purposes. Several times we tried playing what we 
came to call the poker-chip game (e.g., if you had 100 poker chips 
how would you allocate them to the categories in this particular class 
or dimension?) . 



This proved satisfying until we realized that it still did not 
tell us what should be done within each category with the dollars 
represented by the chips. Further, we realized soon that the allo- 
cation of dollars within single classes became immediately suspect 
as we moved to the consideration of allocations within second, third, 
and fourth classes. The relevance of this to building conceptual 
models lies in the fact that one of the classes or dimensions in which 
we tried to play the poker chip game embraced the categories of 
research, development, demonstration, dissemination, training, and con- 
struction. In the context of this paper, the discovery that it was 
no more useful than any other dimension for decision-making means that 
considerable caution should be exercised in using a conceptual model 
based on classification schemes of R&D functions as a basis for R&D 
policy- and decision-making. 

What vje learned is that for purposes of planning and allocating 
resources, it is the problems, goals, and objectives which must be 
firmly in mind, rather than taxonomies. Once the goals and objectives 
are determined (and that is a political and, I would argue, a market 
analysis process) then it becomes possible to derive what the specific 
activities and, therefore, allocations ought to be. 

Lest these comments appear totally destructive of taxonomies let 
me hasten to add that they are still useful heuristics. They help 
to inform policy-makers by insuring that all the options have been 
explored and that in decision-making nothing has been inadvertently 
omitted from sight or consideration. 

The conclusion of this brief discussion of taxonomies, however, 
must be that developing a taxonomy of change processes or research 
functions does not appear particularly useful as a conceptual model 
for research and development if the purpose of the model is to provide 
guidance for top-level policy development, planning, and execution. 

Most research models developed in the past have addressed attention 
to how the several functions identified in various taxonomical approaches 
to research and development relate to one another conceptually, chrono- 
logically, empirically, or administratively. The market model suggests 
that those functions should be viewed in terms bf their relationship 
to increased customer (client) satisfaction. In this light the defini- 
tions and views developed below are offered. 

The key distinction which requires delineation is that between 
research and development, between science and technology. The purpose 
of research or science is the production of verifiable knowledge. The 
purpose of development or technology is the creation of capabilities 
to perform specified functions or achieve specified outcomes where that 
capability did not exist before. Both definitions are broad and there- 
fore require further elaboration. 
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1. Research . Michael Reagan (1967) has effectively disposed 
of the problem of distinguishing between basic and applied research 
in the behavioral and social sciences by the simple expedient of 
denying its meaningfulness, Reagan pointed out the many reasons for 
being dissatisfied with various of the proposed criteria for making 
the distinction. More compellingly , he also demonstrated that there 
is virtually no research which might conceivably be done in behavioral 
and social science which could not be easily connected to a real 
potential policy application of one kind or another. 

If the basic and applied distinction holds little promise for our 
purposes, application of the market concept begins to look a little 
more useful. Basically, it is possible to identify two potential user 
classes (markets) for the knowledge products of research. Ongoing or 
proposed research can be classified into that which will be primarily 
useful to decision-makers of one kind or another. 

The research which is useful to other researchers will tend to 
be theoretically oriented, looking for clues for further research, micro 
rather than macro, interested in exploring a few variables rather than 
many, and so on. For research oriented to decision-makers, it should be 
possible to specify what decisions might be made on the basis of the 
research before it is completed. It will deliberately play with 
many variables rather than a few, be more frequently macro, and so on. 

(An important type of decision-oriented research would be those many 
studies launched explicitly to further the achievement of a specific, 
identified development goal. Each study undertaken within such a 
frame of reference is clearly decision-oriented in terms of the develop- 
ment goal identified.) 

The distinction proposed here is not a new one. The National Academy 
of Education volume. Research for Tomorrow's Schools , (Cronback and Suppes , 
1969) propose that we consider the differences between conclusion-oriented 
and decision-oriented inquiry (pp , 19-27). While they include development 
within the category of decision-oriented inquiry, it makes more sense to 
me to exclude it and preserve the distinction only within the research 
or science domain itself. 

2. Development . The definition of development that I offered above 
is broader than most. Its breadth is a consequence of the basic assump- 
tions which led me to the market model. 

In education we have come to understand that development is a 
systematic process resting insofar as it can on the knowledge produced 
by science. The objective of the process is to create products, materials, 
techniques, and processes which accompish, at an acceptable cost, objec- 
tives specified in advance and deemed desirable or required by learners, 
professionals, or society. These are the usable products of R&D or in 
Levitt’s terms, the things which are designed to satisfy the customer. 
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The broader definition (development as the creation of capabilities 
that did not exist before) is the consequence of important differences 
in the production and delivery of new capabilities in behavioral and 
social technology in comparison to "hard** technology. 

Once a capability is demons trated in computer sciences , electronics , 
aircraft technology, or housing construction, the further delivery of 
the capability is typically one of selling and purchasing a hard pro- 
duct. We know where to go to buy it; once we *ve bought it, it^s ours. 

Social and behavioral technologies are different. They cannot be 
bought in the same fashion. Much more frequently the application of 
social technology means the painstaking development of a new skill or 
technique in a vast army of people who currently occupy the professional 
positions in the social service field in question. On the view being 
offered here, the development process is not completed until the 
requisite skills and human capabilities have been successfully incor- 
porated in the professional repertoires of practitioners. 

In summary, I would make three points. First, reinforcement is 
given the idea that the leading distinction about F&D activities pertains 
to the outputs of those activities .15 

Second, more careful thought needs to be addressed to the expanded 
definition of development. At the present stage of thinking, however, 
it makes good sense to include the development of professional capabilities 
respecting the newly created behavioral or social technologies for 
education as part of the development process . Conceiving of it as part 
of development would assure direct attention to this problem by develop- 
ers and those who make decisions to support development. It also tends 
to underline the critical role of practitioners in the development 
process. Finally, it suggests important modifications which soften 
the unfortunate unidirectional implications of existing concepts of 
diffusion and delivery systems. 

Third, more careful attention needs to be given to decision- 
oriented inquiry for education. My own preference would be the 
broad extension and installation of something like the CIPP model 
of evaluation (context, input, process, and product evaluation) 
throughout the agencies and institutions engaged in or relating to 
the practice of education. This proposition is another way of saying 
building in and using an operations research capability in educating 
agencies of all kinds. Evaluation broadly conceived as in the CIPP 
model is a research activity aimed at decision-makers of various 
kinds and at various levels. The better and more sophisticated the 
inquiry in the name of evaluation, the more useful it is to the 
decision-maker. 



Cf. my "Research, Development, and the Improvement of Education,'* 
Science , Volume 162, Nov, 1, 1968, pp. 541-545. 




Implications for Decision Structures 



The model clearly stresses system or client need over R&D process 
capability. This focuses attention on the importance of collecting 
and analyzing data on client desire and need, societal need, school 
performance, and educational output. Data on these different factors 
then need to be placed in some kind of decision format. 

The specification of educational ends and the allocation of 
educational resources to achieve those ends is a process which 
takes place at many levels in our society. It occurs within schools 
and other educating agencies like the home or the mass media, within 
local and State educational agencies and their governance structures, 
and within the Nation. This process is one in which we all participate 
as citizens, in more or less direct or immediate ways. While as pro- 
fessionals or scientists we may have a role in saying what is feasible, 
in the choice of ends we have no greater or lesser status than any 
other human beings. 

Decision structures are important because they are the framework 
for determining what the desired outcomes of research and development 
activities are or what targets R&D should attempt to reach. (By target 
I mean the production of specific desired end-states for learner-client 
groups or client systems.) The market model requires a considerable 
opening and clarification of the decision-making process in educational 
research from its present relatively closed and muddy state. At the 
Federal level, for example, only a relatively small number of groups 
presently have effective access to the decision-making process. Even 
this is a smaller set than seeks access. The market model implies that 
many groups — those that have been seeking as well as those who have 
not yet been seeking — deliberately need to be brought into more 
formal, substantial, and public processes of decision-making on ed- 
ucational R&D at the national level. The marketplace in education 
is a political marketplace. Its decision structures need to reflect 
more fully that phenomenon. 

What is true at the Federal level is even more true at State and 
local levels. But here the emphasis is somewhat different. If the ends 
educational R&D are serving are political, then political structures 
need to be created or utilized to determine them. But, if the educational 
system is decentralized and the determination of educational policy is 
thus also decentralized, the model suggests that unlike natural or bio- 
medical science, much more attention needs to be directed to the role of 
local and State educational agencies regarding the organization and 
performance of research and development functions. This should extend 
as well to the determination of certain kinds of R&D objectives. This 
is even a more difficult problem than the Federal decision structure 
since at present there is very little of such involvement (with the 
notable exception of the regional educational laboratories) and, perhaps 
even more significantly, there are so few R&D activities to be found 
in State and local agencies to make meaningful involvement even desirable. 
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Finally a third way of viewing the implications of the market model 
for decision structures for R&D is in terms of the transactional processes 
implied by this way of viewing the research and development world. In 
psychoanalytic terms the old ’’father-son” (hierarchical) relationship of 
research to practice (or of therapist to patient) gives way to a negoti- 
ation process of ’’adult to adult” V7here both researcher and practi- 
tioner (or therapist and client) expect to change in significant ways 
as a consequence of encountering one another. While difficult to 
operationalize in the abstract, drawing out the implication in this way 
should lead to interesting and perhaps provocative insights regarding 
the present realities of the relationships currently being displayed 
by participants in R&D and thereby suggest some new directions that 
might profitably be explored. 

The idea of negotiation, bargaining, or consensual processes 
as key element in decision structures for educational research and 
development is a reflection of at least two fundamental propositions 
resident in the model. It is based in a recognition that R&D processes 
are linked to one another in complex ways. Primacy of input from one 
or another of the participants in any linear fashion is virtually im- 
possible to establish. Second, negotiation as a concept also flows 
naturally out of the idea that the practice of research and development 
in education is a social and political process as well as a scientific 
one , 



It is possible to think of several examples of new decision struc- 
tures for research and development which might be constructed on the 
basis of the principles described above. One which appears to have 
some merit and ought to be carefully considered is the idea of holding 
public hearings across the country on a regular basis (perhaps every two 
or three years) in order to secure systematic input leading to decisions as 
to what kinds of development activities to launch. By providing an op- 
portunity for the many constituencies of schools and within schools to 
present their ideas about what is needed, a much more substantial basis 
will have been built for making decisions. The choice of public hearings 
is not a happenstance here, but rather an explicit attempt to employ a 
technique used by political bodies (legislatures) for an R&D area which 
also has strong political dimensions. 

A second possibility is to make much more systematic and open use 
of ’’adminis trative lobbyists . ” While legislative lobbyists are often 
subjected to sharp criticism (and they sometimes deserve it), it is a 
well known fact that our legislative system could not function without 
them. Conceiving of, identifying, and then making use of administrative 
lobbyists might well be a way of assuring a continuing contact with im- 
portant interest groups in education who can provide information and 
expertise and who also need themselves to be informed by scientific de- 
velopments and technical possibilities which seem to be emerging from 
the work now being done. 
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Use of groups like this has been sporadic and unsystematic. It 
has been guided more by vague notions of sharing information (but not 
too much) and trying to "build constituencies." Even the latter aim has 
been more often viewed as necessary because of the threat of their neg- 
ative influence, rather than necessary because research in education is 
of its essence political and therefore linked to the market it is design- 
ed to serve. Using professional organizations, either through their pro- 
fessional staffs or their members, to build their contributions into 
planning and program development in a regular way would be an important 
alteration in existing decision structures. 

A third possibility fully compatible with the two suggested above 
would be systematically to make all program documents associated with 
educational research available to key groups for comment. This would 
apply not only to the Federal program but to State research programs 
and perhaps even the basic program documents of such large research 
organizations as educational laboratories and research and development 
centers. Such procedures would assure opportunities for various con- 
stituent groups to provide information, advice, and feedback. Information 
of this kind can provide useful correctives or materially strengthen 
existing or proposed initiatives. 

A fourth possiblity, more in the nature of feed-forward than feed- 
back, is to take the simple expedient of publicly announcing the dates 
by which certain kinds of decisions have to be made or documents have 
to be prepared. This would provide an opportunity for strongly moti- 
vated or interested individuals and groups to submit data, analyses, 
or proposals respecting the direction and development of the research 
program or its several parts. 

Finally, a fifth possibility is to press for much closer and 
more continual Congressional oversight of educational research and 
development. This idea, advanced several times within the confines of 
the Office of Education during the past three or four years, was 
regularly — I am tempted to say routinely — rejected as likely to 
cause more harm than good. 

Up to the present time Federally supported educational research 
programs receive virtually no public hearing. Appropriations hearings 
in the Senate have tended to deal with the matter in a perfunctory 
manner; in the House the appropriations hearings are held in executive 
session. The current legislation authorizing educational research is 
somewhat unique in having no expiration date, but the consequence of this 
is that the matter does not come up before the substantive co-nmittees 
of the Congres.*; . 

The legislation proposing the National Institute of Education 
offers some promise of change. Public hearings are being held, but 
even the bill as currently drafted makes no provision for regular 
oversight hearings if it should be enacted. Certainly one way in which 
decision structures could be made to reflect more adequately the poli- 
tical dimensions of educational research would be to render the Congres- 
sional oversight function more explicit and more frequent. This 
recommendation will not be popular among^ ,the administrators of research 
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or certain segments of the science community, but it is rather directly 
implied by the propositions underlying the market model. 

The above listing does not exhaust the possibilities, but it 
does give some illustrations of the kinds of techniques which might 
be explored. Key criteria for decision structures designed to reflect 
affinity to the market and the political character of educational re- 
search would be openness, accessibility to all appropriate groups and 
individuals, and responsiveness in the face of that access. 



The Market Model and Institutional Development 

If research managers and policy-makers adopted the market model, 
how might they come to view the needs for institutional development 
of the rich panoply of research and related functions? 

Probably the first realization is that, regardless of the model 
that research managers in the past may have held (if indeed they held 
any at all), what has happened in this country is largely an institu- 
tionalizing of the classic linear model. Like all generalizations 
pertaining to social enterprises, this one can be faulted by individual 
cases here and there. But I think it is nonetheless an accurate one. 

Virtually all of the activities classed as research, for example, 
are to be found in the universities. As a group the Research and 
Development Centers funded by the Office of Education are primarily 
research operations. Some of them, to be sure, have moved to proto- 
type development when theoretical groundings appeared to have the 
strength which would justify such a step. The educational laboratories 
while displaying considerably^ and, to my mind, extremely healthy 
and stimulating diversity, are basically either development institu- 
tions or serving as linkage mechanisms between existing development and 
school and professional audiences. Demonstration activities have been 
the province of Title III of the Elementary and Secondary Education Act 
Thus we have a picture of different stages of the R-D-D process being 
independently institutionalized. 

In practice, of course, there has been no — or very little — 
flow between these institutions to justify and claim that they 
fulfill the requirements of the linear model of R&D. While a couple 
of instances of such flow can be found in ♦‘he relationships between 
R&D centers and education laboratories — notably Pittsburgh and 
Research for Better Schools, and Stanford and the Far West laboratory — 
by and large not much has moved from the work being done by those 
institutions primarily engaged in research, through those primarily 
engaged in development, to those primarily doing demonstrations, and 
finally to operating schools and colleges. In some ways it would be 
surprising were it so, since the programs are so new it would be dif- 
ficult for the relationships to have been established and, in the 
case of Title III and the laboratories, the demonstration projects 
were awarded before the development institutions were even created. 
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If the present situation respecting the institutionalization 
of educational research and development were to be illustrated graph- 
ically, it would look something like Figure 1. Each box constitutes 
a separate institution. The top illustration in Figure 1 fits the 
R6(D models which distinguish the several functions, but do so in a 
nonlinear though linked way. The bottom illustration depicts the 
classical linear model. 




FIG. 1. Two illustrations depicting tha> present (linked and 
linear models) institutionalization of educational 
research and development. 
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The market model suggests a radically different way of conceiving 
(jf institutionalizing research end development functions. Instead of 
defining research functions In terms of their relationship to one 
another, it becomes possible to define them in relation to what their 
outputs are and who might use those outputs. 

The next step is to inquire as to the range of potential users. 

In most instances these are intermediate clients. Immediately we 
can identify State and local educational agencies, colleges and univer- 
sities, educational laboratories, the United States Office of Education, 
the projected National Institute of Education, and so on. Each of these 
institutions, the hypothesis goes, is a market. Each needs to have car- 
ried out by it or for it, in one fashion or another and with greater or 
lesser degrees of sophistication, activities analogous, identical, or 
closely related to the f unctions--research , development , and demon- 
stration — we have come to call research or research related . Graph- 
ically, this would be illustrated as in Figure 2. 



UNIV 



LAB 



SEA 



LEA 



NIE 



FIG. 2. An illustration depicting the presence of 
research , development , demonstration , and 
adoption functions, in several different 
''markets” for educational research. 



A research team, for example, does more than just research. It 
develops research designs or new methodologies. It demonstrates their 
effectiveness by carrying them out, hopefully in a successful manner. 

Development teams do research when they determine what kind of a 
development task to undertake, or what knowledge appropriately under- 
pins the different tasks they might undertake, or when they solve the 
puzzles which continually confront them in the lopment process. 
Demonstrations are important testing exercises for such teams. 



Demonstration programs search out the latest innovations, carry 
out operations research activities to invsure that they have installed 
the demons tration properly , and carry out staff development activities 
to insure that the new programs will function as Intended. 

And operating schools carry out a variety of decision-oriented or 
evaluation research activities. They are continually engaged in stuff 
development, and every successful adoption is a kind of demonstration 
of their continuing capability to meet new challenges and needs. 

Once we accept the basic hypothesis that all institutions or edu- 
cating programs (the ones listed above are only examples) require or 
perform functions analogous to the basic functions indicated in the 
classical research-development-deraonstration terminology, then new 
possibilities begin to open up for institutionalizing systematic 
inquiry in the educational systems of the Nation. 

Earlier attempts to define what was and was not research were 
basically exclusionary in their purpose. The idea was to figure 
out what to eliminate from view as unworthy or inappropriate. The 
view being pressed here adopts the reverse posture. What should be 
included as a legitimate part of a broadly conceived thrust toward 
the establishment of systematic inquiry wherever it might be needed 
in education? The emphasis is on transactional equality rather than 
status, on inclusion rather than exclusion, on the empirical rather 
than the logical. 

Policies respecting institutional development of research for 
education would reflect the decentralized nature of decision-making 
in educational practice. Rather than reserving to the center re- 
sponsibility for the bulk of the decision-making about research and 
development, national research management would seek to decentralize 
power — both authority and responsibility — for major portions of 
the R&D effort. 

This, or course, directly contradicts the basic assumptions 
on which science policy operates today. While allowing argument 
and "partisanship** within the scientific establishment around 
theoretical issues which have not yet been settled, natural and 
biomedical science — which encompasses 91 % of the present Federal 
investment each year in R&D — assume the ultimate ability to dis- 
cover objective truth. It is basically pyramidical and cumulative 
in its assumptions. The operating assumption, therefore, is that 
the best, most up-to-date scientific talent should make the decisions 
as to what research to support. Since the best talent is unevenly 
distributed across the country, the central funding agency is required 
to collect the elite together periodically to make the crucial allocation 
decisions . 

This kind of thinking needs to undergo significant alteration. 

An example of the thinking that needs to be changed here can be seen 
in the recent history of the appropriations for vocational education 
research. In 1968 amendments to the Vocational Education Art required 
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that half of the research appropriation be distributed to t\^e jtates 
on a formula basis. The Bureau of the Budget (BOB) refused to request 
funds under this au'^horization | ostensibly because of the budget squeeze, 
but apparently a majoi consideration in their decision lay In the re- 
quirement that funds should be distributed according to formula. BOB’s 
argument was that research capability was not evenly distributed across 
the country and that to appropriate funds in this fashion was a danger- 
ous precedent. 

Of course, their arguments are based on the characteristics of the 
natural sciences. Equally inportani, in the absence of a rationale more 
suited to the growth and development of the behavioral and social 
sciences, their policy constitutes a classic example of a self-fulfilling 
prophecy at work. It is precisely the correct policy to assure that 
the capability does not develop! 

Attention to the market, to the locus of actual need, will lead 
to policies of institutional development which face the fact of de- 
centralized decision-making. This ought to take at least three forms. 

1. Cne of the strongest needs that the market model suggests 
ought to be served is the cultivation of research techniques associated 
with the identification and definition of op^^ratlonal need. The 
entire range of research activities, most of them probably still 

to be invented, encompassed by the term evaluation or operations 
research needs to be cultivated and encouraged. This cannot be done 
in any other way then as broadly and as uniformly as the institutions 
which need such aid. All of them do. Economies of scale would suggest 
the importance of concentrating first on large State and city school and 
university systems, but the need is equally great wherever formal 
institutions for carrying out educational functions exist. Furthermore, 
the range of techniques and mechanisms ought to be as broad and varied 
as the types of institution*"^ and agencies, and the goals and objectives 
which they are attempting to serve. 

2. A second major thrust would be the deliberate attempt to 
stimulate the creation of autonomous institutions, scattered across 

the country, designed to undertake and carry out their own self-defined 
research and development activities in support of education. The 
educational laboratories were an important beginning at this. The 
attempt should be put back on the original track from which it has 
been shunted as a consequence of executive Impatience, the financial 
squeeze, and mistakiin understandings of role, function, and socis] 
science policy . 

3. Finally, a third policy which would grow out of understandings 
about decentralization and the markets being served would be the con- 
tinual searching out of alternative ways of cultivating research cap- 
abilities. The policy ought to be one of carefully husbanding new 
initiatives, new forms, new techniques. This requires a degree of 
awareness, a kind of patience, and a sense of the long view of things 
difficult to sustain, but, it must be done if the full promise of 
systematic inquiry is to be developed for education. 







The emphasis I have placed here on the decentralizing implications 
of the market model should not be interpreted as being pressed at the 
expense of a strong central influence. The manageii\t:nt at the center 
desperately needs the upgrading of status and role contemplated in the 
proposed establishment of the National Institute of Education, It 
desperately requires the infusion of new managerial and analytic talent 
that the upgrading of status will permit. 

This general elevation of role » however , needs to be undertaken 
with a refined and sensitive understanding of the issues I have been 
discussing in this paper. Nothing would be worse for the projected 
Institute than if it were to conceive of itself as modeled on the 
National Institutes of Health or the National Scie ice Foundation. 

It is in a different business, with different interests, and dif- 
ferent clientele. And one of the most serious missions it will 
have to perform is to figure out just how to develop the best, 
most appropriate mix of work which it performs or derides upon and 
work which it leaves to decentralized agencies to identify and under- 
take . 



Implications for Manpower development 

Two key implications emerge from the market model respecting 
manpower development. The first pertains to the development of the 
skills of systematic inquiry — the research skills, the science 
competencies , the capabilities associated with engineering and 
development. The second focuses on the idea of markets, the way 
in which they are defined, and the capabilities required to assess 
and respond to different responsibilities and incentive or reward 
structures . 

Applied to manpower development policies, it is clear that 
greater efforts need to be directed (£) to the training of people 
to carry out the research responsibilities of operating educational 
institutions and agencies, and (y to developing much finer sensitiv- 
ities of the many different contexts within which personnel trained 
to carry out R&D responsibilities can expect to find themselves. 

The research functions to be performed within operating educational 
agencies are different from those which can be performed in more 
sheltered, university settings. The purposes they serve are different. 
The uses to which the products of such research are put are different. 
The reward structures vary. The markets being served are quite dif- 
ferent and in varying states of sophisticcition respecting such con- 
cepts as the felt/real need distinction or the short-term/long-term 
dis tinction . 

The emphasis to date has been rather heavily toward the academic 
marketplace. And within that domain it has been more toward the 
science side than the technology side. The market model suggests 
greater emphasis on the technology or development side, on the one 
hand, and on the research needs of operating school and university 
systems, on the other. 
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If Education and Science Get 
It Together; A Vision 



At the beginning of this paper I developed the view that models 
are purposive in character and that they depend on the frame of reference 
within which they are formulated, I also suggested that they ought to 
be created interactively with a conception of the goal being pursued. 

This concluding section sets out a vision of that goal. It is a sketch 
of selected characteristics of the educational system as it might be 
if it were integrally related to science and systematic inquiry. 



The Goal 



The goal of the application of science to education is to build 
an educational system (^) whose ends, practices, and structures are 
based upon current knowledge, whose operating philosophy has shifted 
from system maintenance to continuing system renewal, (c) which functions 
within a climate of awareness to desired and actual attainment, and (^) 
which, as a whole, is increasingly accountable to multiple clients, both 
individual and societal. 

The goal statement contains the element of timeliness . It addresses 
directly the question of relevance both for the ends of instruction and 
the means . 

The goal statement refers to the application of current knowledge > 

It therefore implies that the service of science to education is a con- 
tinuing phenomenon, rather than episodic in character. 

The word ^ ^sys tern ** in the definition is intended to focus attention 
on interrelationships as well as elements. This goal statement is de- 
signed to encourage consideration of issues ”up” and **down” the means- 
ends "ladder" (e.g., x is an end but also a means to something else, 
or y is a means, but because it doesn*t exist it is an end we need to 
accomplish) . 

Finally, since systems of education and instruction are never 
content-less, the goal as stated avers that goals and objectives are a 
proper concern of science and systematic inquiry. This is not to say 
that science establishes what the goals or objectives of education 
should be. That clearly is a political or social responsibility. But 
science and scholoarship must be used to illuminate the relationships 
between objectives, the consequenr,es of achieving them, and the fit 
between objectives and the particular means being employed to achieve them. 
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A Scientifically Based Educational System 



If we first assume that major systems for schooling will continue 
to be supported under public as well as private auspices, or at 
least that public support will continue to be available for the purchase 
of educational services of one kind or another, what needs to be done 
next is produce a more finely-shredded "map” — a scenario, if you will — 
which describes in greater detail what a scientifically based educational 
system might look like. In developing these more refined statements, it is 
clearly not possible to predict in detail what the system will be, what 
curricular or instructional objectives it will be serving, or how 
its several parts will interact. But it is possible to develop a vision 
of a selected range of operating characteristics of a scientifically 
based educational system and to express the rationale behind that 
vision . 

Governance, assessment, and accountability . An educational system 
operating on a firm knowledge base and transforming itself continuously 
as that knowledge base evolves would reflect at least two circumstances of 
special importance to governance, assessment, and accountability. First, 
the operating means of the system would reflect the extant and develop- 
ing knowledge base. The techniques of instruction and the organizational 
structures of the institutions established to provide instruction would 
both be based on the most current knowledge about learning, the design 
of systems to foster learning, and the design of organizations and 
institutions to implement those systems. Second, a system based on 
knowledge would display a wider array of alternative forms to express 
the multiple ends the system is serving and the pluralistic outcomes 
desired by individuals and society. 

These two conditions should lead to the development of governance 
structures and incentive mechanisms designed to increase the participation 
of students, parents, communities, employers, and other clients in de- 
fining instructional goals and governing educational institutions. 

More effective and discriminating instruments and techniques for assessing 
progress toward goal attainment would exist. Assessments would be 
available of not only stated goals and objectives but also of the secon- 
dary and tertiary effects of their accomplishment or lack thereof. 

Modified governance procedures will insure the reporting of these assess- 
ments in forms useful and meaningful to client groups. The availability 
of such Instruments is likely to stimulate the development of a much 
broader range of instruction and institutional alternatives than 
exists iX present. 

Rationale ; Goal-setting, assessment, accountability, and 
governance would be important in this system because instructional 
techniques would be effective. That is, since schools would actually 
accomplish what they set out to do, it would make a real difference to 
clients who decide what is going on there. Such a system is also 
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Critics like Ivan Illich, however, raise some provocative questions 
of doubt in this regard. See, for example, his article "Why We Must 
Abolish Schooling," The New York Review of Books , July 2, 1970. 
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likely to be more costly (but more effective). As a consequence 
economically-minded citizens would demand knowledge of results 
achieved for dollars invested. 

In all likelihood, policy-makers interested in majnr improvements 
in schools and colleges would not press directly for change, but 
would pursue the indirect route of examining actual school or college 
output. Confronted with the evidence of performance compared to de- 
sired achievement, teachers and administrators could either be left to 
their own professional consciences as to whether or how to alter 
practices, or public pressures might be mounted which would sooner or 
later produce the intended effect. The availability of effective 
aiid accurate assessment techniques would also be likely to stimulate 
alternative approaches to learning by creating formal routes to 
credentialling through assessment or appraisal which are real alter- 
natives to the current time-serving requirements of schooling. 

A widespread research capability . Assuming no change in the 
political structure of education, research capability would be 
widely distributed throughout the Nation. It would embrace research 
oriented to the improvement of theory and policy, engineering and 
development to build improved learning systems and the capabilities 
in professional personnel to use those systems, and other research- 
related activities. 

Much greater resources, proportionately speaking, would be 
available for research, development, and research-related activities. 
There would be substantially greater sophistication in the. policy 
and management models used to administer R&D. The different purposes 
which could be sem/ed through science would be reflected in a much 
broader array of research and research-related institutions and/or 
functions than presently exists. Thus we would expect to find dif- 
ferent kinds of specialized institutions carrying out research, develop- 
ment, and related activities, but also an array of new functions within 
existing institutions. The research functions undertaken by schools, 
colleges, and other educating agencies would be equally expressive 
of and included under a broadened conception of research and develop- 
ment . 



A much richer variety of techniques, instruments, methodologies, 
and conceptions of educational research would probably be available. 
These would embrace the practice of research and development, its 
management, and the decision structures created to determine its 
policies and directions. 

Finally, the training and development of manpower to perform 
research and related functions would no longer be peripheral to 
the system. It would be as central to the whole educational 
establishment as the training of teachers and administrators 
now is . 
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Rationale ; A science-based educational system will require 
a science establishment and research functions distributed throughout. 

As is the case of the operating system itself, it is reasonable to 
expect, given current trends, that increasing differentiation of task 
and function will be the order of the future here, too. Furthermore, 
as the political and social dimensions of behavioral and social 
science become more clearly understood (and perhaps here we will get 
some help from epistemologists on the nature of knowledge in the 
social domain), it is likely that our conceptions of research and 
development will broaden with a consequent flowering of theories, 
methods , techniques , and approaches . 

Flexible, adaptive organizations , A solid research system producing 
new theories, new knowledge, new ways of doing things, and identifying 
new goals and objectives for education could be justified only if the 
institutions and individuals conceived as potential users of new 
information or techniques were sufficiently flexible, ”aware," and in 
control of their own resources to be responsible and intelligent con- 
sumers and producers of innovation. It seems likely, therefore, that 
a scientifically-based educational system would be comprised of 
institutions which would display rather different characteristics 
than the ones we find at present. 

They would need to possess much more sophisticated techniques 
and staff capabilities for evaluating their own performance relative 
to desired and stated objectives. They would need to become active 
seekers of practices and knowledge to enable them to achieve their 
objectives more efficiently and effectively. They would, in short, 
have institutionalized the inquiry process. 

An output orientation would characterize schools and colleges. 

As a consequence we could expect an increase in the articulation of 
different kinds of institutions and credentialling mechanisms with 
one another in terms of both level and function. 

The system would exhibit much greater differentiation of role 
and function within and between institutions. Such differentiation 
would be accompanied by greater decentralization of authority and 
responsibility for instructional and curricular decision-making. 

Formal institutions of education and learning would exhibit 
much more sophisticated management systems and techniques for ac- 
quiring, processing, and using information in order to be able to 
justify and implement alterations in program. Flexibility would 
appear to require more efficient and sophisticated communications, 
and a capacity for handling relevant information about processes 
and outcomes. Powerful and sophisticated information systems at 
several levels of organization would underlie the desired move to 
flexibility . 
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Finally, flexible and adaptive organizations would contain sub- 
stantial mechanisms for continuing staff development. Some of this 
function might be performed as a consequence of the act of develop- 
ing innovations; the competencies would be engendered by the act 
of creation. Other aspects of staff development, however, would be 
formal training efforts built into the normal working day and ex- 
pected of all instructional personnel. 

Rationale ; To make full use of the products of a solid 
science and research establishment, each part of the system would 
have to know what it is accomplishing, how it is being accomplished, 
and what is required to come ever closer to desired objectives. In- 
formation is seen as the basis, therefore, for flexibility (e.g., 
knowledge of needs and resources is the first requirement for deter- 
mining alternative ways of allocating). 

Information, however, is not the only prerequisite. Attention 
would also need to be directed to how institutions for education and 
instruction can be designed or structured to be able to make use 
of the information which they would have available. Flexible 
structures in the absence of information would tend to fall back 
into the familiar teaching patterns which now prevail. Information 
alone, however, is not likely to free up the eggcrate school or 
the lockstep teaching systems which are currently so prevalent. New 
incentive systems will have to be created and institutionalized as well. 

The current interest in accountability on the part of parents and the 
lay public carries the seeds of the new incentives which are likely to 
be brought to bear. 

The reason why schools and colleges in a scientifically-based system 
are likely to have far different and much more evident staff development 
responsibilities is closely associated with the amount of continuing in- 
novation such a system would experience. Institutions which constantly 
change need to prepare their staffs accordingly. Present practices 
which ostensibly send people elsewhere to receive their training look 
inefficient for such a system, even assuming that such away-from-school 
training could somehow be made more effective than it is currently. 

Linkage mechanisms . An educational system tied to science and 
inquiry would almost certainly possess a continuously updated and uni- 
versally accessible national knowledge bank with capabilities for searching 
for and retrieving data, research documents, reports of good practice, and 
the like. Capabilities for producing literature reviews of individual 
problems, objectives, or policy **targets** would be an important feature 
of such a system. 

Capabilities would be developed to organize, translate, and 
**package** knowledge and tested practices in forms appropriate to the 
needs and characteristics of different potential users. Such ‘^packages** 
might take the operational form of teacher training, demonstrations, 
interpretive materials, consultation services, and so on. 
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State and regional information capabilities would appear, facil- 
itating two-way communication of information about educational needs 
and practices . 

New manpower roles and functions associated with adoption and 
linkage functions would develop and firmly implant themselves as a 
vital component . 

Rationale ; Research on change processes in many fields 
has underscored the importance of linkage mechanisms. These mech- 
anisms provide opportunities for meaningful two-way communication 
among specialized R&D communities, other knowledge producing mech- 
anisms, and operating agencies for the purpose of diffusing know- 
ledge and installing improved practices. I know of no convincing 
arguments why the educational system should be exempt from this need 
and requirement. The application of research to practice is partly 
a problem of choosing the right kinds of research and partly a problem 
of devising usable applications through some kind of developmental 
process. But it is also a problem of making the products of both 
research and invention available to wider audiences than the immediate 
inventors. This is equally true whether the inventions are produced in 
specialized R&D agencies or in operating educational agencies. 



Manpower development . A last set of characteristics focuses 
directly on the mechanisms for training and retraining manpower. 
Conceptually, there are obvious, indeed, already mentioned inter- 
relationships between manpower development and flexible organizations^ 
research and development capability, and accountability, governance, 
and assessment procedures. 

It would be my guess that a rapidly changing institutional 
establishment would almost certainly turn away from college and 
university -based training programs and toward internship, apprentice- 
ship, or on-the-job continuing education techniques for learning new 
roles and functions. The schools themselves will take on training 
functions now being performed by colleges and universities. 

Existing credentialling mechanisms will give way before performance 
criteria. In other words, the educational system will come to apply 
to itself the same kinds of accountability and assessment procedures it 
will increcisingly be asked or caused to accept regarding its performance 
with its immediate clients. The more this happens, the more likely it 
is that schools will become the setting for teacher training and accred- 
itation, both preservice and inservice. 

Lastly, the training of personnel would itself become the subject 
of a considerable amount of R&D. The training of professionals and sub- 
professionals in education would be characterized by much research at- 
tention to learning processes and outcomes and staff development techniques. 



17 Cf. Ronald Havelock, A Comparative Study of the Literature on the 
Dissemination and Utilization of Scientific Knowledge , ED #029 171 




Results would be measured in terms of competencies engendered rather than 
in terms of courses taken or time spent. 

Rationale : Frequent changes in instructional content and 

processes will require changes in manpower and changes in manpower train- 
ing. It seems likely that the same forces which will work on the system 
as a whole would also operate in respect to the development of manpower. 
More attention will be paid to what is needed in the operating system 
than to what is thought desirable by the training system. In other 
words, it seems likely that increased responsiveness of the operating 
system to clients will be reflected inside the system as well. Just 
as externally this will lead to new patterns of governance, internally 
this is likely to lead to quite different arrangements for the training 
of personnel. The investments that school systems make in the training 
of personnel will also be likely to lead them to adopt incentive systems 
for their personnel which will protect the system's training investment 
in them. This is likely to be a further stimulus for role differentia- 
tion and radically revised salary structures. 



Summary 



This paper presents a market model of educational research and de- 
velopment. The model is based on three assumptions: (^) educational 

research is a political and social activity, as well as a scientific 
one; (^) change processes in social systems need to be viewed psycho- 
logically and sociologically rather than mechanically, with the con- 
ditions for change being seen as more important than the content; and, 
(O the political structure of education in this country is, and is 
likely to remain, decentralized. Oriented to the requirements of policy 
development and decision-makers, the analysis addresses the importance 
of identifying and defining client needs first, the availability of 
delivery systems to serve those needs second, and only then what 
innovations or knowledge might be required to create something that 
could be delivered to fill client needs. The market model dissolves 
the status hierarchies for decision-making implicit in linear models 
of R&D and substitutes for them transactional processes among equals. 

The implications of the market model respecting the definition 
of "client" were found to focus on the classical distinction between 
individual and social needs and on the important fact that the consumers 
of innovations produced for the social services are intermediate to 
the ultimate clients. 

The market model causes research functions to be distinguished in 
terms of the outputs of activity and who the users of that output are. 

The implications for decision structures include the development 
of procedures to Ss-cure much greater involvement of clients in deciding 
what kinds of R&D should be done and the invention of techniques which 
better reflect the political character of educational R&D. 






Adoption of the market model would lead to greater decentralization 
in the development of institutional capabilities for educational research 
and development. This would take the form of creating new specialized 
R&D institutions, like the educational laboratories and the inculcation 
of a rich array of research activities in operating educational agencies — 
schools, colleges, universities. State educational agencies, even 
(particularly?) the U.S. Office of Education — designed to inform key 
decision-makers of many kinds about the adequacy of and progress toward 
stated objectives. 

The implied needs for manpower development point squarely to train- 
ing far greater numbers in development, operations research, and evalu- 
ation. The model also suggests the importance of devising ways to bet- 
ter acquaint R&D personnel of the requirements associated with different 
kinds of markets. > 

A concluding section of the paper identifies one vision of the 
general goal behind the integration of systematic inquiry and education 
and suggests what might characterize governance, research capabilities, 
organizational structures, linkage mechanisms, and manpower development 
if the goal were to be achieved. 
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A CRITIQUE OF THE PAPER 



David L. Clark 
Indiana University 



The central thrust of a paper which purports to critique a partic- 
ular framework for viewing educational research, development, diffusion, 
and evaluation should be upon the question of whether the framework, or 
the vantage point provided by the framework, illuminates one’s view of 
the phenomenon being observed. Unfortunately, in terms of simplicity, 
the author has adopted a broad perspective within which to explicate 
his framework and it is impossible to ignore that context in reviewing 
his efforts. This contextual critique is made necessary by the author’s 
insistence on the interdependence and interrelationship of his arguments. 
At the risk, then, of obfuscating the central idea to be critiqued, I 
shall begin at the beginning and proceed sequentially to respond to, 

”A Market Model.” 



A Solution in Search of a Problem 



The author is convinced that educational research and development 
is in ”a muddle.” The difficulty with this muddle is that it is hard 
to define — no doubt that is characteristic of muddles as contrasted 
with problems. However, according to the author, the muddle does have 
certain operational features, to wit: 

. The field of educational research has been in a state of 
political disarray (p. 13). 

The management and administration of educational research 
at the Federal level has been in need of upgrading and 
reorganizing (p. 13). 

. The various constituencies of research in education 
have been hyperexcited , skeptical, horrified, 
hopeful, confused, and groggy (p. 13). 

The proposed solution to the muddle is twofold; first, to turn 
upside down the ”. . . basically linear and hierarchical notions about 
how research and development can improve educational practice” (p . 13); 
and, consequently, second to ”. . . develop a highly sophisticated, 
imaginative, and unyielding concern for the market being served or created 
by educational R&Ef’ (p. 14). 

Even if one were to accept the chain of argument set forth by the 
author in this first section of the paper, and the operational character- 
istics of the muddle the field is in, his conclusions strike this reader 
as overdrawn and indefensible. The problems, you will recall, are political 
disarray, ineffective Federal management, and a bewildered constituency. 

The cmiHu is argued by the author to be the acceptance, and use for planning 



purpoflee, of an Inadequate and unsophisticated framework for viewing 

I I ona 1 R&l) — the so-called linear and hierarcliicai notion. 

To infer tlie cause to be a single conceptual framework, however 
inadequate or misapplied, seems to be an unlikely hypothesis. A much more 
straightforward, plausible alternate hypothesis to account for the problems 
would be, for example, the existence of muddle-headed planner; or, efforts; 
to milk political propaganda advantage from glossy but underfunded programs; 
or, impatience with well conceived progrsims that never had a chance to 
mature and become productive. Interestingly, the author cites exactly 
these alternate hypotheses as he proceeds through the paper and refers 
to (^) ignorance of planners and policy-makers (p . 15), (^) underfunding 
of social science research (p. 16), and (£) impatience with the Regional 
Educational Laboratory Program (p. 31). 

The point to be made is not that one or a combination of the foregoing 
three examples caused the "muddle," but that any one of them is a reason- 
able explanation with as much or more logical support than the reason 
:tted by the author. 

The loosely-stated problem and the unlikely solution do give rise 
to difficulties which are not wholly self-contained in the introductory 
section. The author is much disturbed by the apparently widespread 
acceptance of an othodoxy, the R-D-D-A taxonomy. Many of the criti- 
cisms he directs toward this limited view of educational R&D and 
the misapplication of the vantage point provided by that view (as if 
it were linear and hierarchical) are well taken and will be commented 
upon in more detail later in this paper. However, the conclusion that 
the substitution of a new orthodoxy, the market model, will lead educa- 
tional research from its present muddled state to a productive, visionary 
future misses the whole point of creating and using multiple vantage 
points to make better decisions about research and development in educa- 
tion. Whether the R-D-D-A taxonomy is turned on its head, its tail, or 
sideways, it will still represent a highly limited view of the world 
of educational R&D which, although it may have heuristic value, will 
neither be a sufficient basis for planning or decision-making nor a 
solution to the "muddle." 



Modeling with Feet of Clay 

Most readers who have had experience with the process of model con- 
struction and use in the social and behavioral sciences will have dif- 
ficulty with the peculiar use of the term in this paper. It would 
be convenient to dismiss this problem as nothing more than a pedantic 
question about the use of terms. However, the author is quite serious 
about his use of the term modeling and contends that the "market model" 
is derived from, and based upon, assumptions generated by the modeling 
process . 
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The author is right on target when he notes that the process of 
modeling is designed to increase understanding of a process about 
which less is known than one would wish to know. To achieve this 
better understanding one may turn to a system which is better understood 
and which can be defended as isomorphic to the system cne wishes to 
understand. Isomorphism indicates that the system to be modeled resembles 
the alternate system in form rather than content,. The system to be 
modeled in this case is economic in origin and is noted simply as "the 
market." In fact, the system which serves as the model is explicated 
on pages 19-21 and is based upon one reference in the field of economics 
which discusses the characteristics of growth industries. No effort 
is made to argue convincingly that the original model has either feature 
required of models, i.e.,: 

1. That the system used as a model is understood with 
significantly greater clarity than the alternate 
system under study. 

2. That the system used as a model is, in fact, 
isomorphic to the alternate system. 

Such objections arc neither trivial nor pedantic. If the reader 
is not in a position to be able to identify the elements of the systems 
under consideration, he can hardly assume that relationships held be- 
’tween two elements in one system will hold for similar elements in the 
second system. Yet the author persisuently employs this type of 
argument to support his point of view. 

Modeling is not a self-evidently useful process. As Abraham 
Kaplan points out, where a model is possible but not useful because 
of the crude state of knowledge in a field, modeling may: 

(^) impose premature closure on ideas; and, 

(^) lead to oversimplification or undercomplication 

The reader should, in ray opinion, abandon the notion that he is 
dealing with a model in any usual scientific sense of that term. This 
is not at all equivalent to saying that the reader should ignore what 
the author has to say in subsequent sections of the paper. I am con- 
cerned only that the author's ideas be examined or. their own merits and 
that his arguments be considered aside from the analogies which he too 
often purports are derived by modeling. It is possible to consider, 
apart from the question of the method of derivation, the matter of 
whether he has established a ’'antage point from which can be obtained 
a clear view of educational R&D. 



1 Kaplan, Abraham. The Conduct of Inquiry . San Francisco, California, 
Chandler Publishing Company, 1964, pp. 279-280. 
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ICxainlnlng the AHSumptlons 



If the rc/ulcr ^lecepta the notion of ab/indoning the nw leling thrust 
undcrJylng the paper, he will have to test the basic assumptions with 
groat care since they now form the thrust of the autl^or’s Tirgument. The 
first assumption is that . . science as it is practiced and managed 
in support of education is as much a social and political activity as it 
is a scientific one” (p.l8). 1 am unimpressed by the labored argument 

used by the author to arrive at this conclusion (the interactive effects 
of investigators and variables in physical as contrasted with behavioral 
science studies), but I agree, none-the-less , with the conclusion. The 
simpler argument in support of the point is that the empirical evidence 
of research and development support in education over the past 15 
years demonstrates the assumption as a truism. From the first appro- 
priation in USOE for P.L. 531 in 1956 with its overwhelming emphasis 
on research on the mentally retarded, through the support for the 
National Science Foundation’s Course Content Improvement Program in the 
wake of Sputnik, to the most recent proposal for a relatively inexpensive 
but highly visible National Institute of Education, I think the impact 
of chiefly social and political considerations on Federal level planning 
or ’’scheming” about educational R&D is evident. The author’s more 
involved argument is necessary, from his point of view, since he does 
not agree that past decision-making at the Federal level serves as 
empirical evidence of his assumption. He feels that such planning has 
not taken into account adequately the social and political character of 
science in education. By different routes, then, we have arrived at an 
identical assumption from which we will draw different conclusions. 

The significance of this assumption, once established, varies de- 
pending upon one’s concern for using a framework for educational re- 
search, development, diffusion, and evaluation for Federal level plan- 
ning purposes. The development of perspectives for viewing educational 
R&D, or efforts to reconstruct logically the processes of research, 
developmeit, diffusion, and application in education — or any social 
process field — may be undertaken for a number of reasons. One might, 
for example, be concerned chiefly with a framework which assists in 
distinguishing among behavioral objectives for trainees in R, D, D, and 
A training programs. This author is concerned solely with the develop- 
ment of a framework useful for Federal planning in educational R&D. 

All frameworks or vantage points are not equally useful for all purposes. 
In the context of the author’s concem, the assumption regarding the 
social and political character of scientific activity in education is 
vital. As a matter of fact, in view of the author’s central purpose 
for devising a model to assist in Federal level planning it is nearly 
a self-fulfilling prophecy. 

The second assumption is less easy to accept at face value. Many 
researchers will argue with the statement of the ’’purpose of science" 
which the author states on page 19, i.e., ’’to develop understandings 
or knowledge which hopefully can be applied in some fashion or another 
to achieve desired human ends." The author himself alters this defini- 
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tion on page 28 by noting that, *'The purpose of research or science is 
the production of verifiable knowledge.'* He has, by that time, distin- 
guished between science and technology and notes that the purpose of 
, . technology is the creation of capabilities to perform specified 

functions or a<hieve specified outcomes where that capability did not 
exist before" (p. 28). 

Let us assume then, that the author meant to encompass technology 
in his earlier discussion of the change process in social systems. The 
central thrust of the second assumption is that **. . . the conditi(ms 
which create a readiness in a social field to accept an idea from 
science are more important as far as adoption is concerned that the 
idea itself" or, ". . . in social domains the 'whose time has come* 
part of the phrase is far more important than the *idea*" (p. 19). 

This line of argument is difficult to follow. The author seems 
to be implying that ideas do, in fact, have a life of their own apart 
from Lheir power as ideas or their relationship to science (and, perhaps, 
to technology) . He cites as examples to support this contention, first , 
the nonadoption of strong ideas and second, "faddism" or the adoption of 
low-power innovations . I would contend , as I did earlier in the case 
of the explanation of the **muddle" in which educational R&D finds 
itself, that the atjthor has overlooked more cogent explanations for 
the phenomenon he is examining. Surely it is true that education has 
been characterized by faddism. It is equally true that apparently 
strong ideas have, in many cases, not been implemented. However, the 
dangerously low level of technology or development in the field would 
seem to be a more likely and a more direct explanation than the some- 
what mystical notion that ideas exist on their own in the educational 
ethos. Educators, it could be argued, are fad prone because well 
engineered solutions to their operational problems are not generally 
available. Many low-power ideas are quickly diffused simply because 
they are uncomplicated to apply. Many more powerful notions seem to 
die aborning because they are inadequately engineered. 

This paper has neither the time nor space to develop the foregoing 
argument completely. The only point to be made is that, again, a 
plausible, alternate hypothesis exists which is as well documented as 
the explanation set forth by the author. 

It is critical to note that contrary to the author’s contention 
on page 19, the point in regard to the second assumption is not 
’’simple." The assumption supported by the point, however, is critical 
to the acceptance of the author's model, that is, ". . . the conditions 
causing practitioners and policy-makers to attend to the ideas emerging 
from science must be established parallel to and as part of the support 
of science itself" (p. 19). If the support of educational science 
and technology were at a leve] where well engineered solutions to oper- 
ational problems in schools were available to practitioners in profusion, 
one might have an empirical test of the author's contentions. I think 
it is reasonable to argue at this stage in the development of educational 
R&D that the change strategy implied in the second assumption, noted 



Immediately above in the quotation from page 19, may very well not be 
a necessary component at all; in fact, it could impede rather than facil- 
itate the emergence of ideas **who8e time has come 

In regard to the author’s third assumption, I would debate neither 
the rationale supporting the conclusion nor the conclusion that educational 
decision-making respecting instruction and curriculum will continue to 
be decentralized. The current interest in the administrative organization 
of schools in England and the apparent trend toward further alternatives 
for education in this country can be added to the arguments already 
cited by the ai thor to support his third assumption* 



The Heart of the Matter 



The message of the "market model" is stated in direct fashion on 
page 21: 

"Educational research and development must be conceived in 
terms of the market, consumers, and clients it is supposed 
to serve. Only after that principle is firmly established 
should attention be directed to the processes, techniques, 
and functions which might accomplish that service." 

This statement of principle is introduced by the author with an equally 
unequivocal comment. "From the basic ossumptions and other considera- 
tions developed above .... I am led to one compelling conviction" 

(P. 21). 



The only summary statement I can offer in response is: "I am 
sorry he feels that way." From my previous comments it must be obvious 
that I do not believe that a compelling argument has been built to 
lead the reader to the same conviction as that held by the author. That 
is enough to justify disagreement with the author but I am not. just 
disagreeing — I am lamenting the stated strength of the author’s 
position. I feel it is lamentable because there is the germ of a very 
useful concept in this paper, but the strength of the rhetoric tends to 
obfuscate the concept. 

The author insists on making his point by drawing unfavorable 
comparisons between the so-called "market model" and a previous effort 
to view educational R&D by classifying functions across process 
areas from research through application. This combative tendency has 
caused a substantial loss in intellectual energy. The straw man being 
constructed for destruction by the author is hardly a "bogeyman" at all. 

Perhaps the reader will be assisted in understanding the apparent 
connection between the two frameworks with a brief historical note. 

Some eight years ago this author, along with Dr. Egon Cuba, attempted 
to construct a taxonomy of the processes related to and necessary for 
change in education. This effort, based primarily on sociological 
research on the diffusion process in agriculture and medicine, was pub- 



lishecl formally for the first time in 1967.2 The original taxonomy 
had as its purpose the establishment of an agreed upon set of terms, 
definitions, and objectives which could be used to describe the proces- 
ses necessary to — but note, not sufficient for — change in education. 

The author points out, I think correctly, that this definitional scheme 
is an inadequate base upon which to rest planning for educational R&D. 

He goes on to point out, also correctly in my opinion, that the 
vantage point is not only restricted but basically nonsequential 
and nonhierarchical . This is true and was so noted by the original 
authors. This notation was made explicit to warn readers away from 
applications of the schema which would have been inappropriate. It 
does not, for example, identify the genesis of ideas as being researchers 
rather than practitioners; as a matter of fact, it does not speak to 
that issue because that was beyond its purview. The author notes, 
and this may also be correct, that the definitions have been stretched 
beyond any reasonable limits a;^ a basis for Federal level planning. 

The next step in the author's chain of argument is one to which 
serious objection can be raised. Having noted the inadequacy of any 
model, or in this case a classification schema, to take into account 
or to properly account for all the variables associated with a phenom- 
enon leads neither to (^) abandonment of the original vantage point, 
nor (lb) imperative acceptance of an alternative conceptual structure. 
Knowledge about the change process in education has not reached a 
point where "principles** have emerged or should be established; or, 
where any conception **must** be adopted. This is, as was noted earlier, 
one of the persistent dangers in modeling, i.e., the establishment of 
premature closure on thinking about a phenomenon. 

The reader can, and I would suggest should, ignore the author's 
statement of. imperatives and proceed through the remainder of this 
section as if the author had introduced a new concept for his con- 
sideration. Pretend that the author has suggested consideration of 
how educational R&D might be viewed if the client to be served 
were considered as the heart of the matter; and, if all processes, 
techniques, and functions were organized and carried out to the end that 
the client is able to deal effectively and efficiently with the oper- 
ational problems of education. This is the "big idea" underlying this 
paper and an interesting and useful concept for further consideration. 



Acting "As If" 

The author's "conceptual turnabout" is a mixed bag. I suggest 
that the reader ignore the first half of the section (pp . 21-22) and 
pay close attention to the second half (pp. 21-23). The first part 
of this section is a weak example of modeling. Without reiterating 
the criticisms stated earlier in this paper, note only that the single 
reference employed deals with a system probably not isomorphic to the 
field of education and results in the bad example on pages 21-23 where the 
serv 1 ce to clients is defined as success within the system which is 
supposed to be servicing clients. 



2 National Education Association Center for the Study of Instruction, 
Rational Planning in Curriculum and Instruction , NEA, Washington, D.C. 
1967, pp. 111-133. i 
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Ignoring the model, however, the reader can pick up on an interest- 
ing and fruitful **as if” game initiated by the author on page 23. He 
suggests taking the R-D-D-A "models” (i.e., taxonomies) and creating 
an explicit hierarchy to substitute. for what he feels has been an implicit 
hierarchy. This is his conceptual turnabout and surely does imply a new 
pattern of transaction in educational R&D as well as new ways to 
consider organizing the field. 

One caveat is worth entering at this point. The author notes on 
page 23 that current R&D programs have, in his opinion, been directed 
toward the R&D community as a market rather than toward operating 
educational sites. Most residents of the R&D community will have 
difficulty in swallowing that view. Little has been expended by USOE 
on research training. Individual researchers in all settings are com- 
plaining bitterly that Federal funds to support their efforts are not 
available, and have not been for at least the past three years. Interest 
in science or research, as contrasted with technology or development, 
seems to have been on the decline since the mid-sixties. The reader 
should be aware of the fact that researchers, in general, might well 
contend that the R-D-D-A classification schema, if it ever was used 
seriously for Federal planning, has not resulted in a situation which 
they would feel caters to the R&D community. The point might 
better be made that Federal planning and decision-making has considered 
no market, rather than having favored an inappropriate market. 

The most useful observation the author has to offer prior to the 
delineation of the "model” is encompassed in one sentence on page 23 
when he restates his notion of a transactional process among equals 
(i.e., clients, practitioners, and R&D performers) in resolving 
conceptual problems related to Federal, state, and local planning for 
educational R&D. 



The "Model” 



This is a very difficult section to critique since, in the normal 
usage of the term model, the "model" is missing. Figure 2 in a diagram- 
matic representation of the market model which, predictably considering 
the techniques of modeling employed turns out not to be a model. The 
operational consequence of employing the conceptual turnabout turns 
out to be the notation that all settings are involved, to some extent, 
in all the processes defined in the R, D, D, A taxonomy. As the author 
demonstrates on page 34 and following, this is to some extent true, but 
the examples provided illustrate equally well that the emphases on the 
processes vary sharply and that specialization in function is a natural 
consequence of maintaining differential institutional settings. 

Let me try, as fairly as I can, to note and comment upon the 
points the author cites on pages 34-39 as the heuristics of his model; 



^'Earlier attempts to define what was and was not re- 
search were basically exclusionary in their purpose. 

The idea was to figure out what to eliminate from 
view as unworthy or inappropriate, , , The emphasis 

(in this paper) is on transactional equality rather 
than status, on inclusion rather than exclusion, 
on the empirical rather than the logical'* (p. 37). 

There can be no coui^jj^r argument to the point that definitions are 
basically exclusionary in character. If one attempts to use a word 
to mean something, then it cannot mean everything. The author dis- 
covered this to be true and, consequently, used pages 28-30 to de- 
fine research as contrasted with development so that his readers 
would understand what he was excluding, and including, when he re- 
ferred to research. The earlier efforts to define R-D-D-A were 
directed to broadening rather than limiting the definition of 
processes related to educational R&D. They succeeded, as 
a matter of fact, in focusing attention on such fields as devel- 
opment and evaluation rather than diminishing them or depicting 
them as unworthy. This conclusion by the author is a poorly 
stated derivative of the **model" and is one which tends to con- 
fuse the reader. 

'^Policies respecting institutional development 
of research for education would reflect the de- 
centralized nature of decision-making in educa- 
tional practice** (p. 37). 

As the author rightly points o,ut, the Federal level decision-making 
regarding educational R&D has tended to rely on input from the 
producer rather than the consumer and has tended to assume a pater- 
nalistic attitude toward both, i.e., emphasizing the funder rather 
than either the producer or consumer. Personally, I am sympathetic 
toward the point he is driving home on pages 37-38, but I find it 
hard to relate this point to the model. The argument supporting 
decentralized decision-making was one of the author’s basic as- 
sumptions and the section presented here could have been written 
and supported equally well with no reference to the **model.** 

**One of the strongest needs that the market model suggests 
ought to be served is the cultivation of research 
techniques associated with the identification and de- 
finition of operational need** (p. 38). 

It is true that educational R&D has a remarkably ineffective sy- 
stem of context evaluation. Perhaps the market model would give 
emphasis to this deficiency. 

Another **. . , thrust would be the deliberate at- 
tempt to stimulate the creation of autonomous in- 
stitutions, scattered across the country, designed 
to undertake and carry out their own self-defined 
research and development activities in support 
of education’* (p; 38). 
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How this conclusion is supported by the model I do not know. Since 
there are extant settings and they have, according to the author, 
roles to play in bridging R, D, D, and A, thi^ reference seems anomalous. 
One would guess that the market model would encourage activity closest 
to the point of effective action, i.e., schools and school systems, 
rather than the creation of other autonomous institutions serving as 
intermediate agencies. 

. . the continual searching out of alternative 
ways of cultivating research capabilities’* (p. 38). 

This would appear to be an affective rather than a cognitive problem 
which, hopefully, might be derived from any model proposed to view 
educational R&D. 

**. . . greater efforts need to be directed (first) 
to the training of people to carry out the research 
responsiblities of operating educational institu- 
tions and agencies • . . ** (p. 39). 

A good point and one to which the concept of consumer orientation is 
relevant. The ineffective efforts of USOE to involve public school 
personnel significantly in their Title IV training effort substantiates 
the author's point of view that the planners did not view R, D, D, and A 
training in operational agencies as a high priority concern. 

”... to developing much finer sensitivities of 
the many different contexts within which person- 
nel trained to carry out R&D responsibilities 
can expect to find themselyes** (p. 39). 

A good point similar to the one made above. Most persons, even those 
who would advocate dissimilar models, would support this recommendation. 



An Evaluation of The **Model** 



The model is not a model at all, but instead a single concept. The 
model, as such, does not simplify or make more clear the view of educa- 
tional R&D. It is primarily a suggestion to reorient one^s thinking 
to a view of the client (ambiguously defined) as the focal point from 
which to project an as yet undefined model of educational R&D. 

The author is still functioning chiefly at an exhortative level in the 
area in which he is working. He is disturbed, and I believe rightly so 
that many others concerned with the same area fail to account properly 
for the client in their efforts to construct taxonomies or models. 
Realistically, however, he will have to inject an alternative schema 
with more heuristic value than his present effort, if he expects to 
compete on even terms for the attention of practitioners, researchers, 
and Federal planners. 



58 



Some Random Observations 



The author strokes with a broad brush and, although much of the 
material included on pages 27-34 is not imperative to an understanding 
or critical analysis of his model, he makes it difficult for a critiquer 
to leave unchallenged some of his more general observations on educa- 
tional R&D, 

!• Poker chips . The discussion of USOE’s poker chip game tells 
the reader much more about inept planning than it does about the use of 
taxonomies in the process of planning, A group of planners should not 
have had to "discover*' that planning could not proceed without an ex- 
plication of problems, goals, and objectives. It seems equally strange 
to "discover" that taxonomies are not particularly useful conceptual 
models. One might reverse that procedure and discover that conceptual 
models are not very useful taxonomies. 

The author unintentionally misleads the reader in this section. A 
taxonomy is employed as a straw man to justify the substitution of an 
alternative which needs no justification of this sort. The pressing 
utility of both definitions and taxonomies is immediately made apparent 
by the author himself on pp . 27-28 as he proceeds to define the terms 
which he will employ in his own efforts to construct a model. 

Chicken or egR . The author notes that, "The model clearly 
stresses system or client need over R&D process capability" (p. 31). This 
may well be more a phasing problem than a conceptual distinction 
that makes a difference. If a system has no capability, e.g., no 
trained personnel in R, D, D, A, it may be necessary to emphasize 
the creation of some level of capability before client need can be 
employed for further decision-making. The danger is freezing-in 
some inappropriate R&D capability so that subsequent decision- 
making is based on faulty a priori decisions. This very problem arose 
in the early stages of the USOE*s research training program where 
the press for immediate action caused reinforcement of the extant college- 
based, educational psychology- oriented , training programs at the ex- 
pense of a broader view of the need for trained personnel. This early 
decision-making then became frozen when additional funding for the 
program, which had been anticipated, failed to materialize. 

A balanced view of R&D capabilities and client need might be 
a more reasonable approach than either considered alone or even pre- 
dominately , 

3, The common touch . The author is on very popular ground when 
he contends that ", . , in the choice of ends we have no greater or 
lesser status than any other human beings" (p. 31). Or, "The market- 
place in education is a political marketplace" (p. 31). Although I 
would join with him in insisting that (a^) the political dimensions of 
the educational marketplace have been ignored too often, especially 
by educational researchers ; and, that (^) broader access to decision- 
makings structures at the Federal level would be desirable, I think 
he has become platitudinous in his approach to the problem. Educa- 
tional R&D does operate in a political marketplace, but it also 
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has a significant intellectual content derived from an informed 
scientific community. As the author himself points out, it is possible 
and desirable to gather systematic data on the context of operations 
in education and make this available to decision-makers at all levels. 
Joint recognition of roles in decision-making relating to educational 
R&D might be more helpful at this stage than attempting to redress 
a previous Imbalance by creating a new one. 



Education — A Vision 



No particular end would be served in this critique by commenting 
extensively upon the author ^s vision of the future. For the most part, 
I am, and suspect the reader will be, in sympathy with the points made 
on pages 40-46. The question is whether the model has generated the 
vision or whether the application of the model would facilitate the 
attainment of the vision. Personally, I think neither is the case. 
Education and science have not "put it together." There are many 
reasons why they hav: not. I think few of them are apparent in this 
document on the market model. 



RESPONSE TO THE CRITIQUE 



Hendrik D. Gideonse 



It is disconcerting for a writer to discover that a piece 
of his work has failed almost totally to communicate what he had 
intended. I thought I was writing about the forests we needed; Dean 
Clark found only the trees that we have I 

The only reasonable explanation is that the fault lies some- 
where in my paper. A constructive . response, therefore, is to try 
to restate the case I thought I was making as simply and concisely 
as I know how, but this time in a way which responds to the critique 
that has been rendered. 

Let me start out with an admission of one clear insufficiency in 
my paper. I should have stated the problem I was attempting to solve 
as explicitly as I stated the assumptions underlying the attempt. So 
let me state it now. The problem is the marginal effectiveness of ed- 
ucational R&D relative to its mission, a situation closely related 
to our inability to secure resources on a scale sufficient to the 
task involved. The "political disarray, ineffective Federal management, 
and a bewildered constituency" Dean Clark claims I identify as the 
problem are, or course, merely symptoms of the deeper malaise. 

The paper I drafted was an attempt to develop a prescriptive model 
oriented toward the production of a desired future state of affairs. 

The model was designed not for scientists, but for policy-/and decision- 
makers bearing responsibility for educational R&D. It was designed 
to be used as a conceptual underpinning to help (^) build a more ef- 
fective R&D system for education, (^) build a larger and more effec- 
tive constituency for educational R&D, (O increase desperately needed 
political support, build to a scale of effort which would render 
decision-making about goals and objectives greatly more meaningful 
than it now is, and (^) have, as a consequence of all the above, a 
substantial, continuing, and beneficial impact on the practice of ed- 
ucation in this Nation. 

I italicize the words "prescriptive" and "future" to underscore 
the fundamentally different perception between what I was trying to 
do and what Dean Clark thinks I did. His criticism of ray attempt at 
modeling has basically two related parts: (^) it does not help us 

better understand the empirical phenomena to be found in educational 
R&D; and, (b) it is insufficiently isomorphic to the reality it pur- 
ports to describe. 

As I hope I have now made clear, the market model does not 
claim to describe present reality. Rather, it attempts to pre- 
scribe how we must conceive of educational R&D if the present 
reality is to be significantly altered in desired directions . 

At the risk of incurring a new charge of using an inaccurate analogy. 






die point I am making here is like the idea that all the modeling in 
tlie world of the existing rail and highway transportation systems is not 
going to lead to the design of transportation systems which Involve 
airplanes for speed or pipelines for the transport of bulk commodities. 

1 agree that inappropriate modeling can impose premature closure 
or lead to oversimplification or undercomplication. None of those 
negative consequences, however, can be determined except retrospec- 
tively. If we are permitted to model only for purposes of under- 
standing better the empirical, then we are condemned to cover the 
windshield and drive down the road using only the rear view mi rror I 

As to the problem of isomorphism, it strikes me that the 
economic system is better understood than the educational, and that the 
theories and systems used there are far more powerful in terms of their 
predictabli ty or manipulability than anything we have so far devised 
in education. It is therefore not only legitimate but Instructive to 
employ the analogy and to follow through on the implications of its 
possible application to educational R&D. In any case, it is both 
proper^ and elegant to design a model by considering the function of 
the system being modeled (in this case, serving the education market) 
or what that system is supposed to be accomplishing , i.e., its mis- 
sion, aim, or purpose. 

Having responded to Dean Clark's principal objections, I use 
the remainder of the space available to me to restate my argument 
in simplified form. First, the basic assumptions and beliefs which 
underlie the model. 

My desire to explicate as fully as I could the foundations of 
my attempt at system design was itself a reflection of the idea that, 
in the social domain, belief is oftentimes as important as knowledge, 
and ideas can come true as a consequence of believing and building 
on them . 

Simply stated, my basic assumptions were: 

1. Educational research is inextricably bound up in questions of 
value and is therefore as much a political activity (in the generic sense 
of that term) as it is a scientific one. 

2. Change in social systems is psychological, social, or political. 
It is not primarily logical . It depends on the power of the proposed 
change . 



3. The administrative structure of education is decentralized 
and pluralistic. 




1 Cf. Gerald Nadler, "A Universal Approach to Complex System Design," in 
The Engineering Manager: Survival in the Seventies , 17th Joint Engine- 

ering Management Conference, Montreal, Canada, 9-10 October, 1969, p. 126 
and his "Systems Engineering and Concern for People: Compatible or 

Contradictory," in Technical P^ e rs , AIIE National Conference , 14-16 
May , 1969 , Hous ton , Texas^, p» . 

62 



On the basis of these assumptions and the experience acquired 
over the past six and one-half years participating in the management 
of USOE’s research programs, studying educational R&D, and participating 
in policy reviews (particularly the OECD-sponsored one), I then de- 
veloped a conceptualization of educational research based on its 
function of serving the educational market. This conceptualization 
was stimulated by a paper written by Theodore Levitt in which 
he explored the mechanisms by which major corporations grew and pros- 
pered. His conclusion was that corporations maintained their 
growth position as a consequence of attending to markets rather 
than to products. Levitt concluded that successful industry, be- 
ginning with a customer and his needs, first determines whether 
there is a mechanism for delivering satisfaition, then asks what kind 
of a product needs to be delivered to achieve that satisfaction, and 
finally determines what and whether the raw materials (including the 
knowledge of technical processes, etc.) are available to create the 
product . 

Whether or not the system — in this case, economic — which spurred my 
thinking was isomorphic to educational R&D is finally irrelevant. What 
is important is the suggestion that we have been spending too much of 
our time looking at secondary propositions (how to characterize and 
identify R&D functions and their relationship to one another) instead 
of primary ones (what it was that needed improving, why, and who the 
ultimate beneficiaries of that improvement might be). 

Once the preoccupation with the processes and characteristics 
of educational research is dropped, it suddenly becomes possible to 
see a whole host of things that were not immediately apparent before. 

One of these realizations is that all educational institutions 
perform certain kinds of R&D functions. Of course, what is research 
for one may not be research for another, and the same thing can be 
said for development. That all institutions perform or ought to per- 
form activities which they can legitimately call research or develop- 
ment is an idea that has gotten lost in academic arguments, e.g., 
whether evaluation is really research, or applied research really 
prototype development, and so on. 

The focus on the market also reinforces our understanding of the 
importance of what I called transactional processes among scientists, 
academicians, practitioners, and linkers of one kind or another. This 
bodes well for more effective communication among these many parti- 
cipants, a goal which has often been addressed by educational R&D 
performers and managers, but about which we have not been terribly 
successful. 

A third idea which the market notion reinforces is that decision 
structures for educational R&D should be much more responsive to 
the essential political dimensions of the R&D undertaken. This con- 
cern emerged directly from the basic assumptionr ; but, it was also 
a consequence of the market’s being a decentralized one, and one 
that is characterized by a great deal of diversity. 



7S 



63 



Finally, the market conception suggests that much more needs 
to be done to build the research capacities of operating educational 
agencies. These lie primarily in the evaluation area* The principal 
reason for this conclusion was the need to create a more sophisticated 
market for defining and acquiring R&D products of different kinds* 

All of the implications identified here (there are others and 
they can be found in the body of the paper in the third section) are 
of course policy implications* They are not hypotheses which lead to 
a deeper understanding of what educational R&D now is, but hypotheses 
about what educational R&D must become if it is to be more effective 
than it now is. 

Finally, a remark or two about tone. Of course it was exhortative I 
My purpose was to persuade people of the value of this conception. I 
would go further, however* I would say that one of our great insuf- 
ficiencies is informed passion; education, in its self-consciousness, 
tends to stuffiness, dissection, and dismembering analysis. What we 
need is a little more humor, a little more feeling, and a lot more re- 
sponsible zeal! 



ERIC 
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SYNOPSIS 



The Glass-'Worthen paper analyzes the nature of educational RDD&E 
in relation to the concept of ‘‘disciplined inquiry. “ Within that framed- 
work educational research and evaluation are seen as having the character- 
istics of disciplined inquiry activities, educational development having 
only a few of those characteristics, and educational diffusion having 
none of them. As viewed by Glass and Worthen, educational development 
and diffusion are best considered “inquiry related“ activities. Following 
that analysis the paper focuses in detail upon the nature of and similari- 
ties and differences between educational research and evaluation. 

Disciplined inquiry is portrayed in the paper as a domain with three 
intersecting sub-divisions: historical, philosof' ical, and empiric<^l. 

Educational research and evaluation are differently distributed within 
each; educational development links only to the empirical; and diffusion 
is shown as standing outside all three. Research and evaluation are 
distinguished on the basis of both fundamental properties and strategy 
of implementation; development is distinguished from research and evalua- 
tion by its reliance upon invention and engineering; and diffusion is 
distinguished from all of the above by its identification wita the change 
process (dissemination, demonstration, facilitation of adoption) rather 
than the inquiry process. 

The paper contains recommendations for change In relation to the 
conduct of inquiry-based activities, and a proposal to involve the private 
sector in a way that would Increase the Impact of development and diffusion 
activities on the practice of education. 
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Our intend, is to ex^amine the nature of four inquiry and inquiry- 
related activities in education: research , evaluation , developnient and 

diffusion. Thp relationships among these activities are investigated 
and some proposals for their implementation are presented. The paper 
is organized into six major sections: (£) a cjnsideration of the inquiry 

process and its components; the relationship of research to evalua- 

tion; (jc) the relationship of development co research and evaluation; 

W the relationship of diffusion to other inquiry-related activities; 

(b) proposals for a more useful implementation of educational researcli, 
evaluation, development, and diffusion; and, (O a concluding statement. 
The first four sections are interdependent. The reader who has wearied 
of the tedium of classifications and reclassifications of research , eval- 
uation, development, and diffusion can skip to ^'Proposals ^or a more use- 
ful implementation. . which stands aJone. 



Components of the Inquiry Process 

Research, development, diffusion, and evaluation have been identi- 
fied as classes of activities that collectively represent the spectrum 
of inquiry and inquiry-related processes in education (e.g., Clark and 
Cuba, 1965; Bureau of Research, USOE, 1969; Worthen and Gagne, 1969). 
This conception has also been at least partially verified by empirical 
assessment of functions performed in institutions in which inquiry and 
inquiry-related activities are conducted (Sanders and Worthen, 1970). 

An attempt is made in thi^ section to define and interrelate these four 
classes of activities tc provide a perspective for the analyses and 
recommendations in the remainder of the paper. 



Definitions 

Simple, verbal definitions of research, evaluation, development or 
diffusion are never fully satisfactory. Many recent attempts to define 
these activities treat them as Platonic ideals which would permit simple 
Aristotlian definitions if the definer were only perceptive enough. 

These inquiiry and inquiry-related activities are far too complex to be 
defined adequately as a classicist might have given an acceptable one- 
line definition of a tree. Research, development, diffusion, and eval- 



An earlier draft of this paper appeared under the title, '*Inter- 
reJationships Among Research and Research-related Roles in Education: 

A Conceptual Framework” and was distributed in the AERA Task Force on 
Research Training Technical Paper series. Hcn.’ever, both drafts of this 
paper have been prepared under support of U.S. Office of Education Grant 
OEG-0- 70-4977. 
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uation are complicated endeavors# They are ultimately no more than hypo- 
thetical constructs for a disposition of persons to respond with a certain 
word in the presence of certain instances of the production or utiliza- 
tion of knowledge. The meaning of the words can be seen by examining 
characteristics of the phenomena which elicit from scholars the terms 
researcli, evaluation, development, or diffusion in their writing, their 
ccwersation, and their private thoughts. 

Attempts must be made to differentiate research and research-related 
activities at an abstract, verbal level. It his been argued elsewhere 
that attempting to define and distinguish research and evaluation is not 
mere idleness; confusion of the two accounts for much wasted motion in 
educational scholarship (Glass, 1967). Similarly, failure to recognize 
differences among other inquiry-related activities complicates the 
already difficult job of the would-be trainer of inquii*> and inquiry- 
related personnel in education. In the sections that follow, attempts 
are made to differentiate among these terms, using not only verbal 
definitions, but -also the specification of characteristics that serve 
to distinguish one inquiry- related activity from another. 

In spite of the shortcomings of simple, verbal definitions of 
research, development, diffusion, and evaluation, such definitions can 
serve as a point of departure. The simple definitions that follow will 
be elaborated and defined more fully through the discussion of each 
inquiry- related activity. 

1. Resear :h is the activity aimed at obtaining generalizable 
knowledge by contriving and testing claims about relationships among 
variables or describing generalizable phenomena. 2 This knowledge, which 
may result in theoretical models, functional relationships, or descrip- 
tions, may be obtained by empirical or other systematic methods and may 
or may not have immediate application. 

2. Evaluation is the determination of the worth of a thing. It 
includes obtaining information to judge the worth of an educational 
program, product, or procedure, or the potential utility of alternative 
approaches designed to attain specified objectives. According to 
Scriven (1967): 



"The activity consists simply in gathering 
and combining vf performance data with a weighted 
set of goal scales to yield either comparative or 
numerical ratings; and in the justification of (^) 
the data-gathering instruments, (^ the weightings, 
and (c) the selection of goals" [p. 40]. 



o 

Apologies to mathematics, history, philosophy, etc., for the 
obvious empirical social-science bias in our definition and our thinking. 
We attempt to deal mere with historical and philosophical research in 
the later section on disciplined inquiry in education. 
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3* Development (in education) is the production and testing of 
curriculum materials (including books, films, computer~assis "'^.d instruc- 
tion programs, etc.), organitational or staffing plans (e.g., team 
teaching, differentiated staffing, mocalar scheduling), and other applied 
media or instruments of schooling.^ 

4. Diffusion encompasses planning, designing, and conducting 
activities leading to the application of the knowledge and products of 
research and development. This may be done by various means, including 
(^) the use of communication techniques to disseminate Information about 
the knowledge or product, (h) the conduct of demonstrations to establish 
the utility and applicability of tne product or knowledge, and (O 
procedures which facilitate adoption or application of the product or 
knowledge. 



Disciplined Inquiry in Education 

Cronbach and Suppes (1969) defined disciplined inquiry in education 
as follows: 



"Disciplined inquiry has a quality that 
distinguishes it from other sources of opinion 
and belief. The disciplined inquiry is con- 
ducted and reported in such a way that the 
argument can be painstakingly examined. The 
report does not depend for its appeal on the 
eloquence of the writer or on any surface 
plausibility. The argument is not justified 
by anecdotes or casually assembled fragments 
of evidence. Scholars in each field have 
developed traditional questions that serve as 
touchstones to separate sound argument from in- 
complete or questionable argument. Among other 
things 9 the mathematician asks about axioms, 
the historian about the authenticity of documents, 
the experimental scientist about verifiability of 
observations. Whatever the character of a study, 
if it is disciplined, the investigator has anti- 
cipated the ^ditional questions that are 
pertinent* institutes controls at each step 

of information collection and reasoning to avoid 
the sources of error to which these questions 
refer. If the errors cannot be eliminated, he 
takes them into account by discussing the margin 
for error in his conclusions. Thus the report 
of a disciplined inquiry has a texture that 
displays Ihe raw materials entering the argument 
and the logical processes by which they were 
comp'r^ssed and rearranged to make the conclusion 
credible. * * 



3 

By not defining development as the utilization of results from 
research or evaluation, we have intentionally not prejudiced a later 
discussion of the relationships among research, development, and evaluation. 
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Disciplined inquiry does not necessarily follow 
well-established formal procedures. Some of the 
most excellent inquiry is free-ranging and spec- 
ulative in its initial stages i trying what might 
seem to be bizarre combinations of ideas and 
procedures, or restlessly casting about for ideas 
. . .But. . .fundamental to disciplined inquiry 
is its central attitude, which places a premium on 
objectivity and evidential test” [pp . 15-16 , 18] . 

Inquiry, thus defined, is much more encompassing than the four 
activities of concern here — research, evaluation, development, and 
diffusion — and includes at least three somewhat different types of 
inquiry : empirical inquiry, historical inquiry, and philosophical 

inquiry . Interrelationships showing commonalities and differences 
among these types of inquiry and their components might fruitfully be 
portrayed graphically. Figure 1 is an attempt to depict the domain of 
educational research and research-related activities. The field upon 
which the diagram in Figure 1 is imposed represents language usage (as 
much as anything else) . An element in the field is an instance of an 
inquiry activity being named by scholars either "empirical inquiry,” 
"historical inquiry,” or "philosophical inquiry." The sizes of the circles 
are approximately indicative of the relative frequency of the occurrence 
of the associated terms in intelligent discourse about educational inquiry. 

The interrelationships shown in Figure 1 are obvious. The shaded 
portion is a recognition of other unspecified types of inquiry not 
included under the rubrics "empirical," "historical," and "philosophical." 
The discussion in the following sections is an elaboration of this figure 
and represents the authors* conception of away of depicting, verbally 
and graphically, inquiry and inquiry-related activities in education. 

Empirical inquiry can be thought of as a composite of all areas of 
inquiry where we use observations or experiemnts for purposes such as 
describing existing conditions or verifying or disproving claims, state- 
ments, or hypotheses about relationships among variables. Historical 
inquiry, in general, is the study of development of organizations of 
people (e.g., into cultures or nations) and the study of particular lives, 
movements, or sequences of events. Philosophical inquiry is closely akin 
to rational analysis, loosely based on formal logic and semantics. In 
addition, linguistic analysis and studies of morals and ethics are 
appropriately subsumed under this general rubric. 

In empirical inquiry, the data are generally observations or indirect 
measures of behavior, perceptions, aspirations, and the like. Historical 
inquiry deals more with narrative examination of available evidence (often 
prose of other men) that relates to the historic phenomenon of interest. 
Philosophical inquiry is somewhat like mathematics in that axioms, formal 
propositions , and rules of logic are often the **data** of concern . 

In Figure 1, the areas of overlap are attempts to portray activities 
which draw from more than one type of inquiry. For example, the over- 
lap between empirical and historical inquiiy would include historiography, 
where empirical techniques such as surveys and interviews might be used 
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FIG. 1. The inquiry domain. 
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to prepare a contemporary history (e.g., the history of the IBM Corpora- 
tion or Dow Chemical Company). 

The overlap between empirical and philosophical inquiry might contain 
a variety of activities, such as the use of empirical techniques to stucJy 
various meanings ascribed to l<ey words by persons with differing philo- 
sophic orientation. The use of empirical methods to collect data un 
values of specified groups to enable one to check logical intravalue 
consistency would also be a combination of these two types of inquiry, 
as would also collection of empirical information to determine feasibility 
of alternative goals (which themselves are value-based) in establishing 
educational priorities and objectives. Still another type of inquiry 
that fits here is represented by morals or ethical studies — studies 
aimed at determining what is right or good. For example, one can espouse 
the philosophical view that all men have the prima facie right to live 
and still be faced with a dilemma in operational translation of that 
view when confronted with a greater number of patients who need lieart 
transplants than the number of such organs available for transplants. 
Empirical evidence of other physical corollaries of probable success of 
the operation would be necessary, and one might even argue that each 
candidate's probable benefit to society should be a criterion that would 
require reexamination of the philosophical view and further inquiry of 
this type. 

The overlap between historical and philosophical inquiry might 
well include inquiry into the probable impact of the nonaggressive 
philosophy of Eskimos on the historical decline of their culture as 
contact increased with more aggressive cultures. Conducting an exam- 
ination of sixteenth-century persecution of religious minorities by 
church-influenced sectarian governments as an adjunct to inquiring 
into the appropriateness of maintaining church and state as separate 
might also illustrate a blend of these types of inquiry. 



Inquiry Activities ; Research 
(Basic to Applied) and Evaluation 

Distinguishing between research and evaluation is at best extremely 
difficult and is made no simpler by the fact that each can in turn be 
subdivided into several distinct activities. Many researchers have pro- 
posed their favored schemes for classifying types of research activity 
(e.g., Kerlinger, 1964; Hillway, 1964; and Galfo and Miller, 1970) and 
some have discussed the merits of several alternative classification 
schemes (e.g., Cuba and Clark, undated). Similarly, several contemporary 
evaluation theorists (e.g., Scriven, 1967; Stake, 1967a; Stufflebeam, 
1968; Provus, 1969) have proposed varying types of evaluation. In every 
case the specified research or evaluation activity produces knowledge, 
however general or specific, not previously available. With this rather 
trivial commonality in mind, we turn our attention to th ^ more important 
differences among these ac civi ties . 

The distinction between basic and applied research seeems to be 
well entrenched in the parlance of educational research. Although these 
constructs might more properly be thought of as ends of a continuum than 
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as a dichotomy, they do have utility for differentiating between broad 
classes of inquiry activities.^ The distinction between the two also 
helps in subsequent consideration of the relationship of research to 
evaluation. The National Science Foundation (1960) adopted the follow- 
ing definitions of basic and applied research: 



" Basic research is directed toward increase 
of knowledge; it is research where the primary 
aim of the investigator is a fuller understand- 
ing of the subject under study rather than a 
practical application thereof. Applied research 
is directed toward practical applications of 
knowledge. [Applied research projects] have 
specific commercial objectives with respect to 
either products or processes" [p. 5]. 



Applied research, when successful, results in plans, blueprints, or 
directives for development in ways that baste research does not — this 
point will be elaborated later in the section dealing with the relation- 
ship of research to development. In applied research, the knowledge 
produced must have almost immediate utility; no such constraint is imposed 
on basic research. Basic research results in greater understanding of 
phenomena in systems of related phenomena; the practical utility of this 
knowledge need not be clearly foreseen. 

The opinion is frequently expressed that educational research can 
never be basic research because education is a "practice," unlike psy- 
chology, biology, etc., which are "basic." Such reasoning is mere word 
play, not unlike arguing that basic research in physics is impossible 
because a carpenter "practices physics" when he sets a screw (inclined 
plane) or pulls a tail (lever and fulcrum). Ebel (1967, p. 81) gave 
as one of his reasons for the statement that "basic research in education 
can promise very little improvement in the process of education" that 
"the process of education is not a natural phenomenon of the kind that 
has sonetimes rewarded scientific study in astronomy, physics, chemistry, 
geology, and biology." (Note that in the selection of examples economics, 
sociology, anthropology, etc., were not included.) The English jurist 
Glanville Williams , writing in The Sanctity of Life and the Criminal Law 
(Knopf, 1968), dispensed with "arguments from nature" against contracep- 
tion as follows; ". . .the statement that an act is unnatural, coming 
from a moralist, means little more than that he does not like it" [p. 60]. 
Similarly, Ebel's use of the word "natural" appears to mean little more 
than that he would prefer to exclude education from basic inquiry. 



^Guba and Clark (undated manuscript) argued that the basic-applied 
distinction is dysfunctional and that the two kinds of activities dr not 
rightfully belong on the same continuum. The authors admit to problems 
with this, as with any, classification scheme. However, attempts to 
replace such accepted distinctions with yet another classification system 
seem destined to little more success than discarding the descriptors 
"Democrat" and "Republican" because of recognition of wide variance 
within the political parties. 
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Two activities that might be considered variants of applied researcli 
are “institutional research** and ''operations research,** activities 
directed toward supplying institutions or social systems with data i ilevant 
to their operations. To the extent that the conclusions of inquiries of 
this type are generalizable , at least across time, these activities may 
appropriately be subsumed under the research rubric. However, wliere tlie 
object of the search becomes nongeneralizable infonnation on performance 
characteristics of a specific program or process, tlie label ''evaluation'* 
is more appropriate. 

Evaluation has sometimes been considered merely a form of applied 
research which focuses only on one curriculum, one program, or one lesson. 
This view ignores an obvious difference between the two — the level of 
generality of the knowledge produced. Applied research (as opposed to 
basic research) is mission-oriented and aimed at producing knowledge 
relevant to providing a solution (generalizable) to a general problem. 
Evaluation is focused on collecting specific information relevant to a 
specific problem, program, or product. 

It was mentioned earlier that many “types'* of evaluation have been 
proposed in writings on the subject. Although we are uncomfortable with 
the artificiality of some of the proposed classification schemes, it is 
tempting to include as evaluation activities several widely discussed, 
inquiry- related processes, if only to enrich the abstract construct by 
proliferating instances of the class. For example, the process of needs 
analysis (identifying and comparing intended outcomes of a system with 
actual outcomes on specified variables) might well qualify as an 
evaluation activity if, as Scriven (1967) and Stake (1970) suggested, 
the intended outcomes are themselves thoroughly evaluated. The assessment 
of alternative plans for attaining specified objectives might also be 
considered a unique evaluation function (see the discussion of “input'* 
evaluation by Stufflebeam, 1968), although it might more properly be 
considered a variant form of outcome evaluation occurring earlier in 
the temporal sequence and attempting to establish the worth of alternative 
plans for meeting desired goals. Other proposed evaluation activities 
such as “program monitoring** (Worthen and Gagne, 1969) or *'process eval- 
uation" (Stufflebeam, 1968) seem to belong less to evaluation than to 
operat ions management or some other function in which information is 
collected and decisions are made but no explicit valuation occurs. Of 
course, values are revealed through decisions and choices, but valuing 
and choosing can and should be distinguished, if for no other reason 
than to prevent educational evaluators from turning away from answering 
difficult value questions toward less threatening activites (e.g., needs 
assessment, context description, process monitoring) that are not in- 
tr-Lnsically evaluative. These activities may be attendant to a legiu- 
imate formative or summative evaluation, but they are means of accomplishing 
a comprehensive evaluation and not evaluation in and of themselves. We 
object to the description of context being labeled **context evaluation ** 
or the monitoring of the conduct of a program being called **process eval- 
uation .** Typ-'cally neither contexts nor processes have been evaluated 
when these terms are used; they have simply been observed and described. 

The practitioner may be led by such reckless language into the error of 
thinking he has discharged a responsibility to evaluate his program (when 
in fact he has merely planned it, tried it out, or described it). 



Our concern with whether or not essentially nonevaluative activities 
of the types discussed above should be considered as ‘‘evaluation’* stems 
from the conflict between the roles and the goals of evaluation mentJoned 
by Scriven (1967). Evaluation can contribute to the construction of a 
curriculum, the prediction of academic success, the Improvement of an 
existing course or the analysis of a school district’s need for compen- 
satory education. But these are role s it can play and not its goal . 

The goal of evaluation must be to answer questions of selection, adoption, 
support, and worth of educational materials and activities. It must be 
directed toward answering questions like: “Are the benefits of this cur- 
riculum worth its cost?” or ”Is this textbook superior co its competitors?”. 
The typical evaluator is trained to play more roles than that of simply 
evaluating. However, all of his activities (e.g., test construction, 
needs assessment, context description) do not become evaluation by merit 
of the fact that they are done by an “evaluator.” (Evaluators brush 
their teeth, but teeth brushing is r.ot therefore evaluation.) 

A danger inheres in attempts to develop evaluation models that are 
basically models of the collection of data for decision-making. Such 
models neglect two fundamental points of Scriven's (1969) definition of 
evaluation, viz., that the “activity consists in the . . . combining of 
performance data with a weighted set of goal scales to yield either 
comparative or numerical ratings, and in the justification of (^) the 
data-gathering instruments, (y the weightings, and (O the selection 
of goalt” [p. 40] . 

Decision-centered evaluation methodologists argue that values are 
included in their thinking and their models because a decision is always 
the revelation of a value: if the decision-maker chooses A over B, he 

values A more than B. They believe that values are implicit in decisions. 

For example, Cuba and Stuff lebeam (1968) contended that: 

“The process described as evaluation here comes 
much closer to the root meaning of the term, to 
evaluate, than does the process which currently 
masquerades under the name; we might argue that 
if a name were to be changed it ought to be that 
of present practice. Values come most meaning- 
fully into play when there are choices to be made, 
and the making of choices is the essential act of 
decision-making. What we are proposing here is 
that the entire act of evaluation should center 
on the criteria to be Invoked in making decisions. 

As we shall see, it is through the exposing of 
such criteria that we obtain guidance about the 
kinds of information that should be collected, 
how it shculd be analyzed, and how it should be 
reported. The tem evaluation seems to be 
particularly suited to the process as described 
here, since that process makes such distinctive 
use of value concepts” [p. 28] . 

For a “values-centered evaluator,” however, decisions are implicit 
in evaluation, l.e., in th^ process of measurement against value scales, 
integration of measures into value-^statements , and the justificat -on of 
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the measurement 'and the means of Integrating the measurements, The alter- 
native that scores highest* o*^ a weighted combination of value scales would 
be the preferable altematl/e. However, a decision-centered evaluation 
model can be applied without concentrating attention on the process by 
which a decision-maker integrates Information into an overall judgment, 

Equating values with preferences has precedent in economics. To 
the economist — historicc.lly at least — the value of a product is revealed 
by preferences for it: if the consumer will pay $5.00 for A, then the 

value of A is $5,00. Such a simplistic definition of "value" treats wise 
and foolish evaluation equally; any $5,00 product is as valuable as any 
other $5,00 product. Women regularly pay $5.00 per ounce ( market ^ alue ) 
for a beauty cream, although the constituents— materials and labor--^ 
that cream cont only 25q (the tixie value of the product). That ^ne 
cream can be marketed for $5.00 is testimony to the consumers* Irrational 
belief that expensive products must also be high-quality products. The 
difference between decision-centered evaluation theorists and values- 
centered theorists is the difference between fixing the value of the 
beauty cream at $5.00 because women will pay that price for it and fixing 
its value at 25c because the total investment is a quarter. The analogy 
to educational evaluation is distressingly apt. Administrators have 
been known to choose teaching method A instead of method B, despite 
evaluative data to the contrary or no data at all, because A is expensive. 
The logic runs like this: *'Surely all that expensive gadgetry and those 

priceless materials would not have been produced unless they are an 
improvement over old methods." 

It would be satisfactory to disregard the direct assessment of 
value and merely provide data to decision-makers if decision-makers * 
preferences were always logical, rational, intellig^int revelations of 
value. In truth, most decision-makers ar® perplexed by the decision- 
making process, and many of them rightly feel guilty and insecure about 
their inability to justify their decisions. Hence, it seems unwise to 
view evaluation as the presentation of data to decision-makers who 
must then make the data what they will. 

Evaluation can play many roles in an educational program; it can 
aid the developers by providing mastery test data, it can provide data 
to facilitate administration of the program, etc. However, the goal of 
evaluation must always be to provide an answer to an all-important question: 
Does the program under observation have greater value than its compet- 
itors or sutficient value of itself that it should be maintained? 

Unless inclusion of hybrid inquiry activities becomes essential to 
the point under onsideration, the terms **research" and **evaluation" w"ll 
be used in the remainder of this paper to refer to the "purest** form oi 
each, basic research and outcome evaluation. This approach results in 
oversimplifications, out in discussing the many nuances of meaaing we 
might have obscured Che major points we wish to make. 

In Figure 2, the place of educational research and evaluation in 
the inquiry domain is depicted. The field on which they are imposed is 
the same as in Figure 1 — language usage. An element in the field is 
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FIG. 2. The inquiry domain; 
Educational research and evaluation. 
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nn instance of an inquiry activity being referred to by educationists 
as either **research'* or '^evaluation , " The overlap between research and 
^^valuation is in recognition of commonalities between the two activities 
whiah make it difficult to classify some activities unequivocally in 
either category. 

If the field cn 'hxch Figure 2 is imposed is changed, the relative 
position of research and evaJupiion changes also. For example, if the 
field on which they are imposed consists of techniques and methods, thc’^e 
would be considerably more overlap between these activities than shown 
in Figure 2, since both reseat ch and evaluation are primarily dependent 
upon empirical techniques and methods. However, the two elipses would 
not be coincident, since research and evaluation draw on nonemnirical 
types of inquiry in somewhat different proportions. Educational research 
draws somewhat on both historical and philosophical inquiry. Evaluat‘d on, 
with emphasis upon immediate value questions and no real interest in 
generalizing across time, draws heavily on philosophical inquiry but 
little upon historical inquiry. 

If the field for Figure 2 becomes the purpose for which the activity 
is conducted, there would be little overlap between research and evalua*- 
tion since the object of the search is quite different for these activities 
(as will be argued later in this paper) . 



Inquiry-relaued Activities ; 

Development and Diffusion 

Educational development can be viewed as consisting of three basic 
processes : (a^) inventing the idea or designing the prototype of the 

product, (y engineering and production of the item, and (O tesfiing the 
product in settings like that In which it would be used. Two of these 
processes seem to bo related to inquiry. The invention stage may be 
heuristically related to research in that research findings may stimulate 
ideas for developmental efforts (this thesis will be more fully developed 
later). More directly, the testing stage is obviously evaluation by 
another name. 

Diffusion of research knowledge or developed products includes at 
least three stages: (a) dissemination, (^) demonstration, and (O fa- 

cilitation of adoptionT^ Although all of these stages are necessary if 
research and vievelopment are to influence educational practices, none 
of them is an inquiry activity per se.^ Consequently, diffusion is 

^One of the earliesc and best descriptions of these processes was 
provided by Clark and Cuba (1965) • 

^We have been soundly criticized for taking this position (primarily 
by persons working as diffusers) and have reexamined our position. However 
much we would like to placate our critics, we remain unconvinced that dif- 
fusion can really be thouglit of as inquiry in any meaningful sense. It is 
true that inquiry might be used in relation to such activities, e.g., market 
analysis in identifying dissemination targets. In the vein, inquiry 

can be used in relation to almost any phenomenon, e.g., opinion polls of 
voter preferences in planning campaign strategies. However, such usau'e does 
not qualify politics as an inquiry activity any more than it .Joes diffiision 
as an inquiry activity. 



considered as inquiry- related in that it U’ essential lor implemen- 
tation of the products of research and development activities. 

The relationships of development and diffusion to tii® inquiry domain 
are depicced in Figure 3. 

In the fjllowing sections, iurther attempts will be made to elaborate 
each of the four major inquiry-related activities. Since development 
and diffusion are easier to define and contrast than research and eval- 
uation, we shall deal primarily with characteristics of the latter 
activities in the next major section of this paner, in whijh attention 
is further focused on problems of definition. In the last ha] T of the 
paper — where the iriajor concern is the Implementation of educational 
inquiry — we shall return to development and diffusion. 



The Relationship of Research to Evalu ^n 



In this section we present (^a) a discussion of characteristics of 
inquiry that distinguish educational research from evaluation, (b) a 
consideration of distinguishing features in the implementation of research 
and evaluation, and (^) a summary statement of the relationship of research 
to evaluation. 



Characteristics of Inquiry which 
Distinguish Research from Evaluation 

Research and evaluation have many defining characteristics. Each 
is only Imperfectly correlated with the tendency of Informed men to call 
activity A “research” an.' activity B “evaluation" (just as a clinical 
psychologist uses “anxiety" as a construct to differentiate instances 
of behavior in a way that is not perfectly reproduced by any single measure 
or defining characteristic). The conceptualization of research and eval- 
uation is enriched by the identification of any characteristic of inquiry 
which has a nonzero correlation with thz tendency of intelligent men to 
speak of "research" or "evaluation" when discussing a particular inquiry 
activity. ^ 

Ten characteristics of inquiry which distinguish research from eval- 
uation are discussed below. 

1. Motivation of the Inquirer . Research and evaluation appear 
generally to be undertaken for different reasons. Research is pursued 
largely to satisfy curiosity; evaluation is intended to contribute to 
the solution of a particular kind of practical problem. The researcher 
is intrigued; the evaluator (or, at least, his client) is concerned. 

Although the researcher may believe that his work has great long- 
range payoff, he is no less motivated by curiosity when performing his 



^Scriven (1958, p. 175) referred to such terms as "cluster concepts" 
or "correlational concepts." Such concepts, e.g., "schizophrenia," are 
known by their indicators, all of which are imperfectly related to them. 
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unique function. One must be nimble to nvold becoming bogged down In 
the seeming paradox that the policy derision to support bas^ * Inquiry 
because of Its ultimate practical payoff does no t Iru^ ly that researchers 
are pursuing practical ends In their dally work. Scrlven ( ) has 

argued that* ns regards research In mathematics, practical payoff Is 
increased to the extent that the mathematician is convinced he Is not 
seeking practically significant results. 

The obje ct of the Kesrarrl* aiil evaluation seek different 

ends. Research seeks cone 1 usi onn ; cvaiuacion leads to dec! slons (see 
Tukey, 1960). Cronl)acli miJ Supj'en fl969) distinguished between decision- 
oriented and concluslon-orJ iT.tcd inquiry: 

^’fn a declHlon-orl .ili 1 study the investigator 
is asked to provide liuormation wanted by a 
decision-maker : a school administrator , a govern- 

ment policy-maker, the manager of a project to 
develop a new biology textbook, or the like. The 
decision-oriented study Is a commissioned study. 

The decision-maker believes that he needs Informa- 
tion to guide his actions and he poses the question 
to the investigator. The conclusion-oriented study, 
on the other hand, takes Its direction from the 
investigator's commitments and hunches* The educa- 
tional decision-maker can, at most, arouse the 
investigator's interest in a problem. The latter 
formulates his own question, usually a general one 
rather than a question about a particular institu- 
tion. The aim is to conceptualize and understand 
the chosen phenomenon; a particular finding Is only 
a meqns to that end. Therefore, he concentrates on 
persons and settings that he expects to be 
enlightening" [pp. 20-21]. 

Conclusion-orlcntcd inquiry is much like what is here referred to 
as research; decision-oriented inquiry typifies evaluation as well as 
any three words can. 

3- Laws vs. descrip :: j ons . Closely related to the distinction 
between conclusion-oriented cmd decision-oriented are the familiar 
concepts of nomothetic (law giving) and Idlographlc (deacrlptlva of 
the particular). Research is the quest for laws, i.e., st itments of 
relationship among two or more variables or phenomena. Evaluation 
merely seeks to describe a particular thing with respect to one or more 
scales of value. 

The role of explan^t ion . The nomothetic and idlographlc converge 
ir the act of explanation, viz., the conjoining of general levs with 
descriptions of particular circumstances as in "if you like three-minute 
eggs back home in Portland you'd better ask for a five-minute egg at the 
Brown Palace in Denver because the boiling point of water is directly 
proportional to the absolute pressure (the law), and at 5,280 ft. the 
air pressure is so low in Denver that water bolls at 195 F (the clrcum- 
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stances)." Scientific explanations require scientific laws, and the 
disciplines related to education appear to be far from discovery of the 
general laws on which to base explanations of Incidents of schooling. 

There Is considerable debate among Investigators In education about 
the extent to which evaluatora should explain ("understand") the phenom- 
ena they evaluate. We do rot view explanations as the goal of evaluation. 
A fully proper and useful evaluation can be conducted without explaining 
why the product or program being evaluated Is good or bad or how It 
produces Its effects. This is fortunate, since evaluation In education 
Is so needed and credible explanations of educational phenomena are so 
rare. 



5. Autonomy of the Inquiry . Science as an Independent and 
autonomous enterprise Is such an Important principle that Kaplan (1964) 
wrote first of It In The Conduct of Inquiry ; 

"It Is one of the themes of this book that the 
various sciences, taken together, are not colonics 
subject to the governance of logic, methodology, 
philosophy of science, or any other dlaclpllne 
whatever, but are, and of right ought to be, free 
and Independent. Following John Dewey, I shall 
refer to this declaration of scientific independence 
as the principle of autonomy of inquiry . It Is the 
principle that the pursuit of truth 1^ accountable 
to nothing and to no one not a part of that pursuit 
Itself" [p. 3]. 

Not surprisingly, autonomy of inquiry proves to be an Important 
characteristic for typifying research and evaluation. As was seen 
incidentally in the above quote from Cronbach and Suppes , evaluation 
is undertaken at the behest of a client, but the researcher sets his 
own problems. As will be seen later, the autonomy which the researcher 
and the evaluator enjoy to differing degrees has implications for how 
their respective inquiries arc pursued. 

6. Properties of the phenomena which are assessed . Education 
evaluation is an attempt to assess the worth of a thing and education 
research is an attempt to assess scientific truth. Except that truth 
is highly valued and thus worthwhile, this distinction serves fairly 
well to discriminate research and evaluation. The distinction can 

be given added meaning if "worth" is taken as synonymou with "social 
utility" (which is presumed^ to increase with improved health, happiness, 



^That evaluators must "presume" that certain conditions are worth- 
while and to be sought after leads to what Scrlven called the "polnt-of- 
entry problem in evaluation [personal communication]." Every act of 
evaluation must enter u chain of justification of valued states at some 
point short of the philosopher 'ti rationalization of such elusive concepts 
as "the good life." Educational evaluation can not solve the problems 
of philosophy. Perhaps the best advice for the evaluator is to enter 
at one state above his client, seeking justification for the client's 
definition of "worth" but not for his own. 
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Hfc expectancy, etc., and to decrcaae with increaaefl In privation, 
slrknesH, Ignorance, etc.) and If "aclentific truth" Is Identified with 
two of Its possible forms: (a) empirical verifiability of statements 

about general phenomen.i with accepted methods of Inrpilry; (b) logical 
consistency of such stater.ients . 

Kvaluatlon seeks directly to assess social utility. Research pay 
yield evidence of social utility, but only Indirectly — because empirical 
verifiability of general phenomena and logit. al consistency may eventually 
be socially useful. A touchstone for discriminating an evaluator and a 
researcher Is to ask whether the inquiry would f)e regarded as a failure If 
It produced no Information on whether the plicnomc non studied was useful or 
useless. A research answer researcher will probably say "No." 

In tlip above view. Inquiry la scon as directed toward the assensment 
of three propert.es of statements about a phenomenon: (£) Its empirical 

veri 1 iabl 11 ty by accepted methodn; (^) Its logical consistency with other 
.accepted or known facts; and (O Its social utility. Most disciplined 
Inquiry alms to assess each property In varying degrees. In Figure 4, 
several areas of Inquiry within the field of psychology are classified with 
respect to the degree to which they seek to assess each of the above three 
properties. Tltc distance of a point in the tri.inglc from each of the 
vertices Is inversely related to the extent to which the property 
represented by the vertex Is sought by the particular inquiry. 9 

7. " Universality " of the phenomena studied . Perhaps the highest 
correlate of the research-evaluation distinction is the "universality" 
of the phenomena being studied. (We apologize for the grandness of the 
term "universal" and our inability to find a more hunble word to convey 
the same meaning.) Researchers work with constructs having a currency 
<ind scope of application which make the objects one evaluates seem 
parochial by comparison. An educational psychologist experiments with 
reinforcement" or "need achievement" which ho regaids as neither specific 
to geography nor to one point in tine. The effects of positive rein- 
forcement following upon a response are assumed to be phenomena evidenced 
in most men it: most times; moreover the number of specific instances of 
human behavior which are examples of the working of positive reinforce- 
ment is i^rcat. Not so with the phenomena studied in evaluation. A 
partlculir textbook, an organizational plan, and a film strip have a 
shert li^t expectancy and may not be widely shared. However, whenever 
their cost or potential payof:. rises above a negligible level, they arc 
of interest to the evaluator. 



^Since this conceptualization of inquiry was first presented (Glass, 
1969), the authors have found some interesting corroboration of on authori- 
tative sort. The definition of "theory" in Webster’s Third New International 
Dictionary (1966) is tripartite; "The coherent set of hypothetical, 
conceptual and pragmatic p Inclplcs forming the general form of reference 
for a field of inquiry (as for deducing principles, formulating hypotheses 
for testing, undertaking actions)." The three inquiry activities in 
Webster's definition correspond closely to ti.e three inquiry properties 
in Figure 4, 
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Three aspectH of the general i zabl 1 1 ty ("unlveraal Ity") of a phenomenon 
(an be Identified: (a) generality acroaa time (Will the phenomenon— 

e.g., a textbook or a particular "self-concept" — be of interest 50 
years henc«?); (b) generality across geography (Is the phenomenon of any 
Interest to people in the next town, the next state, across the ocean?); 
and, (c) applicability to a number of specific instances of the general 
phenomenon (Are there many specific examples of the phenomenon being 
studied or is this the "one and (^nly"'!). Tiiese three features of the 
object of an educational Inquiry can be 'jsed to (lasslfy different Inquiry 
types . 



T>irce types ol inquiry are ic pi e'4(’nL»'d in Finote 5: (ij) program 

evaluation--the evaluation of a complex oi i>eople, materials, and 
organization which make up a particular educ.it 1 onal program; (b) product 
evaluatlon--the evaluation of a medium of schooling such as a book, 
a film, or a recorded tape; and, (/ ) educational research. 

Program evaluation is depicted as concerned with a phenomenon (an 
"educational program") which has limited general izabillty across time 
and geography; the innovative "ecology curriculum" (including instructional 
matcri.ils, staff, students, and other courses in the school) In the 
Middletown Public Schools will probably not survive the decade, is of 
little Interest to the schools In Norfolk which have a different set of 
environmental problems and instructional resources, and has little rela- 
tionship to other curricula with other objectives. Product evaluation 
is concerned with assessing the worth of something like a new ecology 
textbook or an overhead projector which can be widely disseminated 
geographically but which similarly will no', be of interest 10 years 
hence nor does the study of Its properties produce any reliable knowledge 
about the educational process In general. The concepts upon which 
educational research Is carried out arc supposed to be relatively per- 
manent, applicable to schooling nearly everywhere, and they should 
aubsune a large number of Instances of teaciilng and learning. 

8. Salience of the value question . At least In theory, a value 
can be placed on the outcome of any inquiry, and all inquiry Is directed 
toward the discovery of something worthwhile and useful. In evaluation 
it is usually quite clear that some question of value is being addressed. 
Indeed, value questions are the sine qua non of evaluation and usually 
determine what information is sought. This Is not to say that value 
questions are not germane in research. The goals may be the same for 
both research and evaluation — placing valuii> on alrcmntive explanations 
or hypotheses — but the roles, the use to which information may be put, 
may be quite different. For example, the acquisition of knowledge and 
improvement of self-concept are clearly value- laden objectives. The 
value question in the derivation of a new oblique transformation technique 
in factor analysis is not so obvious, but it is there nonetheless. Our 
purpose in raising tU..s point lb to call attention to the fact that with 
respect to assessing the value of things, the difference between research 
and evaluation is one of degree, not of kind. 

9. Investigative techniques . A substantial amount of opinion has 
been expressed recently to the effect that research and evaluation should 
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FIG. 5. Three inquiry cypes classified by the 
"universality” of the phenoaenon investigated. 

(The property represented by each axis is absent where the 
three axes ocet and increases as one CK>ves out along each axis. 



«-f';p)oy different techniques for gatlierlng and processing data, that the 
t^M'ilindR appropriate to research--sucli as comparative experimental 
'!(•'* I ;',n---are not appropriate to evaluation, or tliat wltli respect to 
ti'< III IqiK's of empirical Inquiry, evaluation la a tfilng apart (Cronbacli, 
Carroll, 1965; Stufflebeam, 1968; Gulia and f;tuf I lebea/n , 1968). 
lor ex/itnple, Cuba and Stufflebeam wrote: "On the surface, the applica- 

tion of experimental design to evaluation problems seems reasonable, 
since traditionally both experimental research and evaluation have been 
used to test hypotheses about the effects of treatments. However, tliere 
-ire . . . distinct problems with this reasoning" (p. I'i). Most of tlie 
alleged problems, hen^ever, stem from Cuba and Stuf f lebeam ' s conception 
<)( the comparative experiment. They asserted that for comparative designs 
to yield v.'lld results: 

". . . the treatment and control conditions must 
l)e applied and held constant tliroughout the period 
of the experiment, l.c., they must conform to the 
Initial definitions of these conditions. The new 
or traditional program conditions could not be 
modified In process, since In that event one could 
not tell what was being evaluated" Ip* 13]. 

Cuba .and Stufflebeam appear to conceive of "treatments" as being so 
narrowly .md strictly defined that they permit decision-makers no free- 
dom to adapt and modify treatments as they are being applied. Surely 
sucli confining treatments arc not required for valid experimental 
comparisons . 

An educational treatment may simply create an identifiable context 
wltliln which decision-makers .ire free to adapt the program to the exigencies 
of the moment. A medical researcher evaluating a drug against a placebo 
Is free to administer other substances to control side-effects or to vary 
the amount of dosage In accord with his observations of the progress of 
remission of the disease. Minor modifications during the course of an 
experiment do not destroy the validity of the drug vs. placebo comparison 
since they are often a necessary part of the context which is being eval- 
uated, namely the "treatment of Disease X by Drug A." The decision- 
maker could, of course, so alter the context of the application of a 
treatment that the originally defined treatment Is no longer being eval- 
uated, as for example when a medical researcher stops administering the 
experimental drug because serious side-effects show up In the patients, 
in sudj instances, the opportunity to evaluate comparatively has simply 
been lost. However, that some comparisons may not be feasible does not 
make all comparisons Impossible. 

Cuba and Stufflebeam also maintained that "... all students in 
tlie experiment must receive the same amount of the treatment to which 
they are assigned . . ." (p. 13]. Comparative experimental designs 
require no such thing. Here and above, Cuba and Stufflebeam appear to 
conceive of a "treatment" as a fixed entity in nature like a quart of 
ml^k or an aspirin. A "treatment" in a comparative experiment is usually 
an .abstraction- a construct — ^with defining characteristics which 
create a context; the context created is all that one can evaluate. The 




context seldom demands that .ill experimental subjects receive the same 
amount of something, nconomlsts ran experiments on the "negative Income 
tax" In New Jersey in the Into 1900 's: persons on a negative Income tax 
plan were compared with pirsons on the conventional IRS plan on variables 
like unemployment rate, woik incentive, spending and saving habits, etc. 

The very essercc of the negatlv- it.ciTre tax is that its amount varler. 
from person to person, yet no one claims that the comparison is thereby 
invalidated. Indeed, not all subjects need even receive the san.e thing , 
as when we evaluate Individualized instruction. 

Cuba and Stufflebeam claimed that the application of comparative 
experimental design to evaluation problems ". . . conflicts with the 
principle that evaluation should facilitate the continual Improvement 
of a nrogram" and that ". . . it is useful for making decisions after 
a project has run full cycle but almost useless as a device for making 
decisions during the planning and implementation of a project" (pp, K,1S]. 
These criticisms were reuolved earlier by Scriven's (1967) distinction 
between formative and summative evaluation when Cronbach (1963) raised 
the same points. 

Comparative experimental design by Cuba and Stufflebeam was faulted 
because of the "near impossibility" of controlling or eliminating "con> 
founding variables" through randomization or otherwise. Cronbach (1963) 
raised the same point: "Any failure to equate the classes taking the 

competing courses will Jeopardize the interpretation of an experiment 
and such failures are almost inevitable" (p. 676]. "Equation" of groups 
is an impossibility recognized early in the history of experimental design. 

In comparative experimental design groups are made "randomly equ: valent" — 
which is not strictly equivalent at all — and postexperimental di. ferences 
are inspected to reveal whether they are amall enough to be attr i>uted to 
the original random assignment or whether a treatment effect mu£'. be pos- 
tulated to account for the large difference. Thus, valid experl lental 
comparisons are not impossible Just because experimenters cannot perfectly 
equate groups. Valid, probabilistic comparisons are possible, the grow- 
ing number of well-designed comparative experiments in education demonstrates. 
It is true that valid experimental designs are difficult and expensive to 
implement: but educational evaluators have yet to learn that such designs 
arc usually worth the cost. 

Finally, Cuba and Stufflebeam (1968) wrote that: "A fourth problem 

inherent in the application of conventional experimental design io the 
possibility that while internal validity may be gained through the control 
of extraneous variables . such an achievement is accomplished the 
expense of external validity " [p.16). Internal and external validity 
are not Incompatible. Designing experiments which evidence both types 
of validity' to a high degree is simply a technological problem in 
instrumentation, data collection, and statistical analysis (see Bracht 
and GlatiS, 1968). 

While there may he legitimate differences between research and 
evaluation methods, we see far more similarities than differences between 
research and evaluation with regard to the techniques by which empirical 
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evidence 1b collected and Judged to be sound. Ab Stake and Denny (1969) 
f nd 1 rated ; 

"The dlellnctlon between rcBearch and evaluation 
can be overstated as well as understated . . . 

Researchers and evaluators vork within the sane 
Inquiry paradigm . . . (training programs for} 
both must Include skill development In general 
educational research methodology" [p. 374). 

Hftiphlll (1969) expressed the same opinion when he wrote: "The 

conceq !<!nce of the differences betweet. the proper function of evaluation 
studio and research studies Is not to be found ... in the methods o' 
Inquire of the researcher and of the evaluator" (p. 220). We shall 
returr in a later section to the question of how research and evaluation 
differ In the techniques of Investigation they employ. 

'’cc notion that evaluation Is really only sloppy research has a 
low Incidence In writing but a high incidence in convernation — usually 
among icsearchers but also among some evaluators. This bit of slander 
arises partly from a misconstruing of the concept of "experimental control." 
One form of experimental control is achieved by the randomization of 
extraneous influences in a comparative experiment so that the effect of 
an Intervention (or treatment) on a dependent variable con be observed. 

Such control can be achieved cither In the laboratory or the field; 
achieving it is a simple matter of designing an internally valid plan 
for assigning experimental units tc treatment conditions. Basic research 
has no proprietary rights to such er lerlmental control; It con be attained 
in the comparative study of two reir forcement schedules as well as in the 
comparative study of two curricula. 

A second form of control concerns the ability of the experimenter 
CO probe the complex of conditions he crertea when he intervenes to 
set up an "Independent variable" and to determine which critical element 
in a swarm of elements Is fundamental in the causal relationship between 
Independent and dependent variable. Control of this type occupies the 
greater part of the efforts of the resear her to gain understanding; 
however, It is properly of little concern to the evaluator. It is 
enough for the evaluator to know that something attendant upon the 
Installation of curriculum A (and not an extraneous, "uncontrolled" 

Influence unrelated to the curriculum) Is responsible for the valued 
outcome; to give a more definite answer about what that something Is 
would carry evaluation into analytic research. Analytic research on 
the nongeneral izable phenomena of evaluation is seldom worth the expense. 

In this sense only Is evaluation "sloppy." 

10. Criteria for Judging the activity . The two most Important 
criteria for Judging the adequacy of research are internal validity 
(to what cuctent are the results of the study unequivocal and not con* 
founded with extraneous or systematic uncontrolled variance) and external 
validity (to what extent can the results be generalized to other units 
persons, classrooms, etc.). If one were forced to choose th4'-two 
most important from among the several criteria which might be used for 
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Jiidgifig the adequacy of evai ual 1 on , they wcnjld probably be la ^..jrphlsm 
(t<i what extent Is the Infomatlan obtained Isomorphic with the reality- 
based ‘nfornatlon desired) and rredlbillty (to what extent Is the informa- 
tion • lewed a?, believable by <llents who need to use the Informal icm) . 



Dlst Inguls h lnfi re atu r es In the 
Implem enta tion of Researc h an d Eval uat Icjn 

There la a point to our concern with the abstract nature of different 
Inquiry activities. It may seem that too ouch Is be'ng made of Idio- 
syncratic definitions of "research" and "evaluation." We can.iot help 
being greatly concerned with the meanings of words and, more Importantly, 
with bow they Influence action. The meanings which "research," "eval- 
uation," "development," and "diffusion" hold for a person make practical 
differences In how he will pursue Inquiry. Often the most sincere and 
energetic efforts at inquiry are misdirected because of a semantic 
confusion. By happenstance, habit, or methodological bias, the trial 
and investigation of a new curriculum or organizational plan will be 
labeled an "experiment" Instead of an "evaluation." The inquiry carried 
out Is different for having been called an "experiment" and not an 
"evaluation." The choice of language predisposes the literature that 
the Inquirer rcadt (It will deal with .experimental design), the consultants 
he uses (experts in designing experiments) . and how he reports the results 
(always in the best tradition of the J :njmal of Experimenta l Psychology ). 

In some instances, none of these paths will lead to the relevant data 
or a ‘cnslSie evaluation. The crucial data may be "soft" instead of 
"hard"; the object of the search may he valuations instead of explanations. 

Alstract meaning can give direction to practical decisions; the 
cxai.inctlon of intelligent practice lends meaniig to abstractions. Thus 
wc dwell In the next few pages on two major practical Impllcationa of the 
conceptn of Inquiry preaented thus far. 

1. Dlaclpllnary baae . The call to ,ouike educational research 
mult Idiacipllnary is good advice for the research community as a whole; 
but the individual researcher la ill-advised to attack his area of 
interest simultaneously from several different disciplinery bases. 

Some few men can fnilt fully work in the cracks between disciplines; 
but most will find it challenge enough to deal with the problems of 
education from the perspective of one dlsclpine at a time. The effort 
required to master even a small comer of a discipline works against 
making Leonardos of all educational researchers. 

That the educational researclier can afforo to pursue inquiry within 
one paradigm and the evalusv'on team cannot is a consequence of the 
autonomy of Inquiry. When one is free to define his problems for inquiry 
(as the researcher is), he seldom asks a question that takas him outside 
of the discipline in which he was trained. Psychologists pose questions 
that can be solved by the methods of psychology. The seeds of Che answer 
to a research question are planted with the question. The evaluation 
team enjoys less freedom In Che definition of the questions they oust 
answer. Hence, the answers are not as llkaly to be found by use of a 
traditional methodology. Typically, then, the evaluation ten finds 
it necessary to employ a wider range of inquiry perspectives and tech- 
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to with the they rpfiolve.^^ (See 

different rolen wtilth reHe.ir' liei ^ ;itid evn‘pint<it H 
plriv reqnire llflerent training program?^. Five elements of graduate- 
(<vel 1 raining pr^^gr^ns for resea tellers and for evaltiators are contras ted 
Ut'] nu , 

<al IH r, f i p I i nan' bases. Feaeartb aired it produ^in)* *, under- 
standing of ?4u^li plien»>nena as Pchc»ol Jearnlngi t h*- sf^rla] o rgani 7 ..it Ion 
'd '^^honls, liuman growth rnd developmi^nt , etc., < an bent he arrompllshed 
Iron, the persper fives afforded by r he traditional soc. i al -s <: I cures . Ihc 
best training fc>r many eduration/il researchers Is llkelv to he a tliorougli 
'tar. j c ry of n relatively narf'^^w, tr a«ll t ic»nal , social aclcncc discipline. 

Wv regard as mis taken tlit r.otlon of a unique science of education con- 
oirurte*] c?ut of unique r»dnt at 1 ona 1 fonstrufts with methc;dB of inaulry 
whirl) are not al ready a part of tlie traditional social science. 
l'uy<hology, sociology, etr., are not so lacking In conceptual richness 
that they cannot be made to accominod.atc most of the phenomena of education. 
7he fact that relatively few researchers trained In these disciplines 
oxlilblt Interest In working on problems with apparent applicability in 
e*ju(atlon dnen not negate tlie argument. Katt)er, It merely suggests 
that existing reward structures lead to most researchers conducting 
inquiry within their home discipline. One proposal for training would 
ho to recruit vigorously among bachelor and master’s degree holders in 
tiu'sc behavioral and social sciences to attract them into our educational 
re?u*arch programs. Then, their training could consist of continued 
preparation In their discipline coupled with application of the tools 
ol tlnit discipline to educ.ntlonal problems. Wc would anticipate that 
pr»rfu)ns trained In tills m.-inner would do much to improve educational 
rese.arch by bringing tlic perspectives of the disciplines to bear. 

Ihls docs not .argue that any one researcher should receive training 
In mor^ th.an one discipline. A research tr.tlning program that attempts 
to train the same student In social psychology, physiological psychology 



The discussion has taken on an utopian tone. In reality much 
ov.'iJuailon Is becoming .is stereotyped as most research, tvsluatcrs 
often take part In .'tsking the question they will ultimately answer, 
tnd tliey are all too prone to generate questions out of a particular 
"evaluation model" (e.g., the Stake model, tlie CIl*r modal) rather than 
a "discipline." Stereotyping a method threatens evaluation as it 
threatens researdi (Cla;4S. 1969). 

^^This does not mean tliat it Is inappropriate to continue to train 
research methodologists wi thi n the field of education, "Educational 
research me thodology " Is an aggregate of techniques borrowed from agronomy , 
psychology, mathematics, etc., with some refinements of educationists. 

*b sons trained in tills area play critical loles In (a^) developing and 
reiJnlng statistical teclinlques and (M advising othe7 educational 
researchers (whose strengths lie raoie in thel»* discipline backgrounds 
than In their knowledge of methodology) in matters of design, statistics, 

t' 1 1' . 







and micro- economic theory is doomed; the result will be a researcher so 
superficially trained in se/eral areas that he can contribute in none. 

By contrast, to the extent that the training of evaluators touches 
on the traditional disciplines at all, it is best that several disciplines 
be sampled. The trainee should appreciate th'" ^/iew of education afforded 
by each of the socially-relevant disciplines. In this way, the evaluator 
can become sensitive to the wide range of phenomena to which he must 
attend if he is to properly assess the worth of an educational program. 

That superficial exposure to the many social Sciences does not train 
him to build a science of education is simply irrelevant. 

(^) Methodological training. The evaluator's role is largely 
that of a methodological expert applying inquiry techniques to the 
solution of a particular type of practical problem. Consequently, emphasis 
in trair*ing must be given to such topics as statistical analysis, 
measurement and psychometrics, survey research methods, and experimental 
design. The techniques — in this sense — that a single evaluator 
might use are little different from the body of techniques of the 
empirical social sciences. (We confess no sympathy with the argument 
that such techniques as inferential statistics and comparative experi- 
mental design are not useful in evaluation, even though we agree that 
they are often misapplied. Cuba and Stufflebeam, 1968, argued that 
comparative experimental design is inappropriate to the purposes of 
educational evaluation; an attempt to rebut the argument appears in 
Glass, 1969.) 

The training program for the educational researcher need not be 
as broad in techniques of inquiry as that for the evaluator. Most 
sociologists' work is none the worse for their ignorance of confounding 
relations in fractional factorial designs, and who would argue that an 
experimental psychologist suffers from his ignorance of scalogram analysis? 
The training of researchers must be more concerned with substantive matters 
(e.g., Keynesian economics, Skinnerian operant conditioning, and cognitive 
dissonance theory) than with methods and techniques per se. 

(O Organizational and management training. Most educational 
research projects, funded or not, usually Involve few persons— of ten 
only one. Such activities demand for their directors only minimal 
organizational or managerial skills. E. L. Thorndike probably never 
resorted to a PERT chart in any of his researches on the law of effect, 
for example. 

It is difficult to dete mine whether educational research is 
typically small in scale because it is educational or because it is 
usually psychological. Sociologically based educational research, e.g., 
Coleman and Campbell's Equality of Educational Opportunity (1966), can 
become organizationally complex and impossible without great attention 
to management. However, such research seems to be the exception rather 
than the rule in education. In short, while all researchers could 
benefit exposure to concepts of project management such as PERT or 
critical path analysis, relatively few researchers will need extensive 
preparation in project organization and management. 
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Many evaluation efforts are essentially **onc-man 8hows**--e .g . , 
evaluation of many ESCA Title I and Title TIL projects fall into this 
category. However, many new evaluation units in public schools, state 
departments of education, and the like comprise a larger number of 
workers than most research projects. Also, the techniques used by 
evaluators arc likely to be more diverse than those used by the uni- 
diccipiinary researcher. As a result, many evaluators in education 
may profit from more extensive training in management techniques. 
Organizing the efforts of large numbers of workers and managing diverse 
activities associated with evaluation clearly demand high level skills 
in mcmagement, especially If one adopts a systems or **team” model of 
evaluation such as that proposed by Provus (1969). Ralph Tyler surely 
operated with the 1930 's equivalent of the PERT chart in directing the 
Eight Year Study, one of the earliest formal educational evaluations. 
Given the scope of some current evaluation efforts, even more management 
sophistication seems to be required. 

(^) Practicura or internship training. Some provisions in 
training both researchers and evaluators must be made for the acquisition 
of practical experience to complement course work. Doubtless there is 
no better practical experience for the research trainee than apprentice- 
ship to a competent, practicing researcher. Worthe.i and Roaden (1968) 
found that there is a significant relationship (p .001) between 
genuine research assistantship experiences and subsequent productivity 
as an educational researcher. Moreover, it was found in a later study 
(Worthen and Roaden, 1970) that apprenticeship to one researcher over 
an extended period of time spent working on a variety of projects was 
positively correlated with subsequent productivity (p <T .025) whereas 
an assignment to work on a single, isolated research project was not. 

Internship experiences in public schools, state education depart- 
ments and similar "front-line" institutions are probably unproductive 
in the training of researchers. Few researchers' capacity to produce 
good research has been enhanced by the firsthand knowledge that school 
administrators are subjected to political pressures, that junior-high 
teachers seldom have planning periods, or that students are too pre- 
occupied to learn during the last two months of their senior year. 

The internship experience "in the field" is entirely a d:?fferent- 
matter in the curriculum of the evaluation trainee. A protracted 
apprenticeship to a single researcher would be Inappropriate in the 
training of the evaluator. Breadth of experience in a variety of settings 
is important. The researcher can afford to ignore — indeed, must ignore 
— the countless practical constraints of contemporary schools in con- 
structing his elegant idealizations. Evaluation that ignores practical- 
ities is just bad evaluation. 

(je) Selection of trainees. Con’^ldering how little genuine 
change takes place in students even in intensive graduate-level train- 
ing programs, the selection of trainees is as Important as the experiences 
that comprise the programs. Logical criteria for trainee selection 
which are consistent with our conceptions of research and evaluation 
can be identified. Empirical validations of the logical criteria are 
lacking. 
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Doth evaluators and researchers should be selected from among the 
most intelligent products of undergraduate education. The notion that 
researchers should be more intelligent (in the Graduate Record Examina- 
tion sensei say) than evaluators Is an entirely unjustif if ble and repug- 
nant value Judgment. Maturity and teaching or comparable educational 
experience Is not irrelevant In the selection of evaluation traineesi 
though neither should be overstressed. The Irrational imposition of 
these criteria in the selection of educational researchers works against 
securing the best candidates. The pattern of Interests and values 
most conducive to success as an evaluator may occur at a slightly greater 
rate among the graduates of professional colleges (engineeringi educatloni 
etc.) than among the graduates of liberal arts curricula. Conversely | 
there may be some coincidence between an undergraduate liberal arts major 
and an attraction to the life of uninterrupted scholarship so essential 
to the researcher In any field and so rare. 



The Relationship Between Educational Research 
and Evaluation; A Summary 



Evaluation borrows Inquiry techniques and a knowledge base for 
recognizing value from research and contributes little in return. Method- 
ological research in the social sciences produces the technology of data 
collection and analysis that are so Important to empirical educational 
evaluation. In additioni knowledge produced by basic research is often 
critical in determining whether a particular finding from an evaluative 
study should count as good or bad. For example, the evaluative meaning 
given to the finding that a health curriculum decreased the incidence 
of cigarette smoking among teenagers derives from extensive medical 
research which established a causal link between cigarette smoking and 
cancer and coronary disease. 

The view of the relationship between research and evaluation 
presented here is a parasitic one; others such as Suchman (1967) see the 
two living symbiotically: 

”To some extent evaluative research may offer 
a bridge between ”pure” and "applied” research. 

Evaluation may be viewed as a field test of the 
validity of cause-effect hypotheses in basic 
science whether these be in the field of biology 
(l.e.i medicine) or sociology (i.e., social work). 

Action programs in any professional field should 
be based upon the best available scientific know- 
ledge and theory of that field. As such, evaluations 
of the success or failure of these programs are 
intimately tied to the proof or disproof of such 
knowledge. Since such a knowledge base is the 
foundation of any action program, the evaluation 
research worker who approaches his task in the 
spirit of testing some theoretical proposition 
rather than a set of administrative practices will 
in the long run make the most significant contri- 
bution to the program development" [p. 1970]. 
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Our entire discussion of the distinction between research and 
evaluation (and the relationship between research and development developed 
later In this paper) runs counter to Suchman’s ^'Islon of the same rela- 
tionship. Suchman saw ^action programs** as based on scientific knowledge 
and theory; we see a more tenuous link between (a) what can be conceived 
of abstractly and established empirically In the laboratory and (b) what 
can be implemented In the field. There can hardly be said to exist an 
**lntimate** connection between the success of a program in the field 
and some theory or hypothesis from a basic discipline. 

The course of most basic research leads It further away from the 
possibility of Influencing practice directly. Basic research seems to 
be characterized by a succession of studies in which greater control 
(**control** In the sense of the ability to manipulate specific components 
of Independent variables) Is exercised at each stage so that relation- 
ships among variables can be more precisely determined (witness the 
course of learning psychology from the Law of Effect through complex 
unravellngs of the nature of reward, the function of drive states, 
secondary reinforcement, etc.). The control required to achieve theo- 
retical understanding (e.g., the necessity to employ nonsense syllables 
in much research on memory and learning) leads to the **lack of relevance*' 
of basic research so widely decried by practitioners. The practitioner's 
disappointment In the "irrelevance" of basic research is matched by the 
researcher's dismay at testing any "theoretical proposition" In the 
welter of uncontrolled variables that exists in even the simplest "action 
program." 



The Relationship of Development to Educational 
Research and Evaluation 



The dependence of development upon sophisticated evaluation tech- 
niques is obvious; indeed, development was earlier defined as consisting 
of production and testing of various educational products. That this 
critical component of development — evaluation — is often omitted 
from the development process is clear; the field of education is replete 
with examples of random adoption of faddish, untested innovations. In 
such cases, the process of development is incomplete and, by rights, 
those who promulgate such innovations (whether the motivation is profit, 
notoriety, or sincere belief) should be viewed as skeptically as the 
sideshow barker and other species of huckster. The need for careful 
and thorough evaluation (call it field study, product testing, or what- 
ever) as a part of the process of development is well established; the 
fact that this relationship is so plain as to appear simplistic should 
not deter us from devoting serious attention to seeing that it is imple- 
mented in practice. 

The relationship of development to educational research is more 
complex. This may lie partly in the fact that current conceptions of 
educational development (including our own) are still in the formative 
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stages^ making the identification of relationships with other activities 
very difficult. The complexity is increased by the fact that temporal 
relationships between research and development are not as clear as some- 
times supposed. 

The implications of scientific knowledge from basic research for 
educational devv>lopment depend on the generality of the former. The 
most useful scientific laws are those of physics, most of whi:h are 
learned firsthand and nearly unconsciously early in life by everyone. 

These basic physical laws are nearly perfectly generalizable ; within 
human experiences there is no evidence that they have ever been suspended 
on the surface of the earth at any time in history. An astronaut in 
his spaceship experiences the repeal of certain physical lavs and must 
laboriously learn a new set. By contrast, the laws of the social and 
behavioral sciences are of extremely limited generality. The facts (and 
their scarcity) made the ready application of social science knowledge 
to education extremely difficult. Social scientific laws are little 
more than probability statements about the tendency of one variable to 
be related to another, 'hese laws are highly interactive with factors 
such as geography, time, cultures, and individual characteristics. The 
most comprehensive and sophisticated behavioral theory of the last 100 
years seems to continue to lose validity as mankind moves further from 
the society of sexually repressed, 19th century Viennese hysterics toward 
the contemporary model patient, the compulsive neurotic* If physical 
laws were as limited in generality as the lavs so far discovered about 
the social system, we would hesitantly creep out of bed each morning 
not knowing whether we would float to the ceiling or crash to the floor. 

If basic scientific knowledge from educational research cannot be 
directly applied to educational development, hov then are basic and 
practical knowledge related? There are two views on the problem: the 

metaphoric and the heuristic . 

One position that is gaining advocates is that the lawful relation- 
ships discovered in basic inquiry serve as metaphors in the minds of 
schoolmen for the reality of education. This is the conception of the 
relationship between conclusion-oriented and decision-oriented inquiry 
advanced by Cronbach and Suppes (1969) and Jacob Getzels in an address 
delivered at the Annual Meeting of the American Educational Research 
Association in February 1969. Getzels argued that research may have 
the greatest effect on education not so much when it attempts to alter 



^^In our opinion, the content and skills that comprise educational 
development will continue to depend upon appeals to authority until 
empirical investigations of the type currently being conducted under 
the Oregon Studies in Educational RDD&E and the research project of the 
AERA Task Force on Research Training are completed. 

^^Schutz, one of the leading thinkers and practitioners of develop- 
ment has recently noted that: "Educational development is at present 

in a state of conueptual deprivation" [196P, p. 3]. 
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the elements of classroom practice directly as Indirectly when it contributes 
to transformations in the general paradigms and conceptions of the human 
being (AERA, 1969). The metaphoric view holds that the practice of 
education is ditermined largely by a "prevailing view" In the minds of 
schoolmen and the community which Is a metaphor of systems of scientific 
knowledge in psychology, sociology, political science, etc. The 
scientific knowledge .and the folk knowledge are loosely related; the 
one system Is only a metaphor of the other. The paths through which 
basic knowledge affects the prevailing view are not well understood, 
but they surely Include mass media, textbooks, Journalistic writing, 
and the oral tradition of education. 

The metaphoric view of the relationship of basic research to practice 
is essentially the position described eloquently by Boulding (1956) In 
Ills book The Image: Knowledge In klfe and Society , which Is required 

reading for all worker’s In the knowledge Industry. The metaphoric view 
Is probably true as a description of how schoolmen have used scientific 
knowledge In the past. However, we cannot accept It as i necessity nor 
as a prescription for the relationship of educational research and 
development In the future. 

Metaphors are too Inexact to serve as reliable guides to practice. 

Where the truth of half of the metaphor is distorted In traiismlsslon 
to the prevailing view or where the scientific basis of the metaphor 
Is technical or subtle, the chances are too great that scientific truth 
will lead metaphorically to practical error. 

In addition, the lag time between the discovery of a fundamental 
fact and Its inclusion 1 p the prevailing view is Intolerably long. It 
Is sobering to realize that only a small fraction of the children in 
this country have ever studied under programmed instruction and that 
it will be decades before many teachers will have a rudimentary grasp 
of operant learning principles. Schoolmen's resistance to new knowledge 
contributes greatly to the lag time between discovery and appll -.atlon. 

Finally, trusting the improvement of education to schoolmen's 
metaphoric interpretations of scientific knowledge Is dangerous because 
of the way scientific knowledge grows. As basic inquiry in any area 
comes of age, the sophistication and complexity of the knowledge produced 
increases. Doubtless, no physical law discovered In the last 60 years 
has affected the way we use physical knowledge daily. Chemistry no 
longer speaks to layman and the audience for the findings of experimental 
psychology is dwindling. As a science grows t becomes more complex, 
more subtle, and less able to influence the "prevailing view" (or, in 
the present case, less able to influence educational development). Vlltness 
the fact that in the early days Thorndike, Thurstone, and others were 
able to apply mental test theory directly in test construction; but,, 
recent research in psychometrics is increasingly less immediately 
applicable . 

Consider the example of laterality ("handedness") and learning 
difficulties. Until recently in this country (and currently in oarts 
of Ireland), children were punished for writing with their left hand. 
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Ancient prejudice’s against left-handedness (e.g., the literal meaning 
of the Latin word ’’sinister*' is ’’left-handed”) and a resulting tendency 
for teachers to ascribe the learning failures of left-handed pupils to 
their laterality instead of stupidity or poor Instruction supported the 
prevailing view in schoolmen's minds that pupils should learn to write 
with their right hand. The research which shows no relationship between 
laterality and learning problems is now more than 30 years old; but 
it is written in n technical language few ochoolmen understand. 

The 1 ag time , the opportunity for m ltrinterpretation t and the 
divergence of scientific and po pular images make the metaphoric use of 
educational knowledge by schoolmen an inef ficient and dubious activity . 

Fundamental knowledge might better be regarded as having a heuristic 
relationship to educational development. The word heuristic is used 
here in the literal sense, namely suggesting or stimulating empirical 
research. The impact of basic educational research will be more Immediate 
and reliable if educational researchers regard each new finding as a 
stimulus to a developmental-evaluative (and, perhaps, an applied research) 
effort. Basic knowledge will never prescribe £ particular development , 
but it will stimulate creative minds who understand the basic knowledge 
to develop materials and practices based loosely on it . If the innovation 
is developed and eval uated with the best empirical methods of evaluation 
and applied research ., the payoff from basic research can be realized 
more quickly at a f uller value . 

One of the more interesting areas of basic inquiry in education 
in the last 10 years is the work of Rothkopf, Frase, Anderson, and others 
on ’’mathemagenic" behavior, in particular the control of attention. 

These researchers have demonstrated the striking effects on learning of 



^^Here, we seem to come full cycle. Development might be seen as 
drawing heuristically on basic inquiry. However, it also leads logically 
to applied research in that the nature of the product being tested may 
well result in a level of generallzable findings more relevant to applied 
research than to the typical evaluation study. For example, field test- 
ing Copper Valley School District's newly-developed, space-related science 
curriculum is clearly an evaluation activity; generalizability of the 
evaluation findings across time, geography, and to related phenomena are 
of little interest. Conversely, field testing the Far Ncrth Regional 
Educational Laboratories' newly-developed approach to teaching "mini- 
concepts” might be considered applied research in that the generaliz- 
ability of the findings across geography, time, and to specific instances 
of the phenomenon are all of interest. Also, applied research may well 
supercede evaluation as products become more formalized and appear in 
semi-final form. Although evaluation studies are critical components 
of developmental research, the concern is typically with the single case 
or group under consideration, and unless evidence of the worth of a new 
procedure generated from a "single case” evaluation study is supported 
by replicable evidence, then support for the new procedure cannot be 
generalized except through appeals to authority. Thus, basic research 
might (very loosely) lead to development, which In turn necessitates 
evaluation and possibly further (applied) research. 
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ihe control of attention through prompts In programmed instruction and 
adjunct questions in textual materials. On the latter question, it has 
been demonstrated that questions preceding paragraphs to be read cue 
the learner's attention to the answers to the questions and away from 
incidental material. Postparagraph questions cannot cue specific 
passages in a paragraph, and thus result in greater acquisition of 
information. In programmed instruction, overprompting of stimulus and 
response in associative learning has been shown to cause inattention 
to the stimulus^response pairing, resulting in poor learning. 

The conclusions these researchers have reached concerning the cuing 
and control of attention are as well understood and lawful as basic 
educational research can hope for. Yet further work in this area has 
shown the facilitative effect to be highly interactive with such factors 
as positioning and pacing of questions or prompts, the difficulty of 
the material to be learned, the propinquity of question and answer, 
whether questions are general or specific, whether subjects can turn 
back in a text for review, habituation to adjunct questions, the moti- 
vation of the learner, amount of awards for learning, meaningfulness 
of material to be learned. Immediate or delayed retention, and the 
mode (visual or auditory) of presentation of the material. The mathe- 
magenic effect can appear and disappear as a function of these conditions. 
Obviously, then, simple prescriptions for curriculum development or 
instructional management cannot be made on the basis of these findings. 

The effective utilization of this basic knowledge without an intermediate 
applied research study would depend upon: (^) the developer's ability 

to recognize or measure such elusive characteristics of instructional 
materials and the conditions of their use as degree of meaningfulness, 
incentive-generating properties, and pupils' self-selected pacing and 
order of study; and, (^) the invocation of ceteris paribus in the hope 
that nothing will disturb the fragile basic relationship on which the 
developmental effort is based. It seems vain to hope that basic know- 
ledge will be implemented without applied research and evaluation to 
test under special circumstances that which basic research only suggests 
may be true. This point was made in the following somewhat murky state- 
ment of Merton (1949): 

"(£) Every applied research must include some 
speculative inquiry into the role of diverse 
factors which can only be roughly assessed, not 
meticulously studied. 

(^) The validity of the concrete forecast depends 
upon the degree of (noncompensated) error in any 
phase of the total inquiry. The weakest links in 
the chain of applied research may typically consist 
of the estimates of contingent conditions under 
which the investigated variables will in fact operate. 

(O To this degree, the recommendations for policy 
do not flow directly and exclusively from the 
research. Recommendations are the product of the 
research and the estimates of contingent conditions, 
these estimates not being of the same order of 
probability or precision as the more abstract inter- 
relations examined in the research itself. 
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((d) Such contingencies make for indeLerminacy of 
the recommendations derived from the research and 
thus create a gap between research and policy*^ 

[p. 1761.15 

For some time to come, education Is more likely to be improved 
through Ingenious Invention Inspired by basic research and hard-headed 
evaluation of the Inventions than by the direct application of the fruits 
of basic educational Inquiry* 



The Relationship of Diffusion to 
Other Inquiry-related Actlvltles -^^ 

Diffusion Is perhaps more clearly Identified with "educational 
change" than Is any other activity related to Inquiry* To be surci 
students of educational change take Into consideration educational 
research, development, and evaluation; but, most of their attention 
has focused on diffusion processes* For many years, cultural anthro- 
pologists and rural sociologists were In the vanguard of thinking about 
change processes. Much of their Interest was In diffusion rates of new 
Ideas and practices In agriculture and among primitive peoples* Mort 
(1962) was among the first to study diffusion rates of practical Inven- 
tions In education* In recent years, persons In many other disciplines 
(e*g*. Industry and the military) have pursued Inquiry In this area. 
Havelock and Guskln's (1969) comprehensive review of literature on 
knowledge utilization processes Is evidence that an impressive body of 



^^Thls quote Is an excellent synopsis of several of the points made 
In this section* However, It would be a distortion of Merton’s over- 
all position on the relationship between "pure" and "applied" research 
to Imply that the position taken here and presented earlier by the authors 
(In the section on the relationship of research to evaluation) coincides 
with Merton's* Here we have argued for a much looser link between research 
and development than does Merton* Earlier we presented nearly the opposite 
view on the relationship between research and evaluation from what Merton 
advanced (1949, p. 179)* 

^^Dlf fusion, development, evaluation, and research are all obviously 
related In that each activity can be applied to the others* One can do 
research on diffusion, development, or evaluation processes* Similarly, 
one can (^) evaluate research, development, and diffusion, (^) develop 
materials to train evaluators, diffusers, and researchers, and (O diffuse 
the results of research, development, and evaluation activities* We 
find such relationships obvious and conceptually barren and apologize 
for building this section around the relationships listed In "c" above* 
However, diffusion seems to us to be a special case that Is related to 
the other activities In only this way: we reiterate that we do not see 

It as an Inquiry activity per se* The more meaningful Interrelationships 
that appear among the other activities (e*g*, the areas of overlap In 
Figure 3) simply are lacking when diffusion Is considered* 
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literature la accumulating In this field. However, despite tlie array 
of Information that can be martlaled In support of the Importance of 
diffusion activities, two problems remain. First, the agrarian model 
of the change process Is pervasive to the point where It Is often Inap- 
propriately applied In contexts quite different from that In wlilch It 
was developed and should be used. Second, a majority of our knowledge 
about diffusion stems from position statements of authorities In the 
field. The relatively small number of studies In this area seem to focus 
either on (£) surveys of opinions about diffusion processes. Innovations, 
etc., or (b) studies of the rate of diffusion of various Inventions and 
products. As a result, there exists a fair amount of knowledge about 
how fast diffusion occurs and what barriers often Impede It; but, precious 
little is known about how diffusion actually occurs, or what specific 
knowledge and skills are essential for adequate performance of diffusion 
loles. This Is especially troubling In view of the current pressure to 
train large numbers of diffusion personnel to serve "linkage" roles in 
education. Urging such training seems presumptuous In the absence of 
evidence that such efforts would in fact help to "bridge the gap" 
between research and practice (we will return to develop this point 
more fully later in this paper). Persons responsible for major decisions 
regarding allocation of funds for training inquiry-related personnel 
might wisely defer any sizable investments in training diffusers until 
more is known about the role they are expected to play. 



Diffusion of Research Knowledge 



Science could not long function without adequate exchange among 
scientists of information resulting from research activities. Indeed, 
the exchange of knowledge is directly related to the raison d'etre of 
research — the production of new knowledge. Research is interdependent 
and Incremental In that every study Is related to and builds upon extant 
knowledge; this relationship cannot exist unless there is efficient 
diffusion of research findings among members of scientific communities. 

Traditional models of diffusion of research (e.g., journal articles 
and papers read at formal conferences) are woefully ineffective, and 
new information exchange systems (e.g., ERIC) have proved disappointing 
to date. Efforts to develop more effective mechanisms to disseminate 
scientific knowledge should be encouraged;^® perhaps the day will come 
when one no longer learns most of what he knows about new developments 
In his field through Informal and haphazard contacts with colleagues. 



^^The deplorable time lags and inefficient mechanisms currently in 
use in relation to at least "educational science" suggests that either 
the position above is overstated or educational science is functioning 
marginally, if at all. 

^®Perhaps one promising model is that proposed by the American 
Psychological Association in their new Experimental Publication System 
which features reviews' of manuscripts within two days of receipt and 
distribution to all subscribers in topical areas within 65 days of the 
receipt of manuscripts. 
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Support for more effective ways to disseminate scientific knowledge 
docs not imply support for disseminating a greater bulk of information. 

The authors agree with Michael’s (1963) statement that: ’’Probably, on 

the average, only 10 percent of published papers in educational Journals 
are worthy of being reported in the Review [ of Educational Research ] ” 

(p. 443]. Scriven (1960) also set the figure as low as 10%. The over- 
burden of useless information resulting from poor research caused 
Asher (1969) to argue that research utilization should include not only 
development, dissemination, and adoption, but also an elimination function 
by which the profession could rid itself of such static in the communica- 
tions system. We do not intend here to explore ways to Identify and 
exclude knowledge that is not worth disseminating, but it is obvious 
that solving this problem would facilitate better storage, retrieval, 
and dissemination of valuable knowledge. 



Diffusion of Evaluation Findings 

Diffusion plays a limited role in evaluation since evaluation 
findings and recommendations based on those findings must be disseminated 
to clients, constituents, etc. However, dissemination of this type is 
purely local in nature and is likely to be specific to each combination 
of setting and type of evaluation. Diffusion of evaluation findings 
could be best performed in most Instances by the evaluator or his staff, 
with the help of communications consultants when necessary. 



Diffusion of Products of Development 

The most widely discussed role of diffusion in education is in 
facilitating application of the materials, devices, systems, or methods 
produced through educational development activities. This type of 
diffusion differs from those discussed above in two ways. First, it 
Includes all three processes subsumed under diffusion — dissemination, 
demonstration, and facilitating adoption. Diffusion of research and 
evaluation findings is concerned only with dissemination of knowledge 
or Information. Second, exchange of money is much more relevant in 
diffusion of developed products than it is in pure diffusion of know- 
ledge. The importance of this distinction will be expanded later in 
this paper. 



Toward More Useful Implementation of Educational 
Research, Development, Evaluation, and Diffusion 

So far, we have discussed research, evaluation, development, and 
diffusion in abstract terms, with little regard for the realities of 
how these activities are initiated, organized, and supported. In this 
section we shall discuss federal policy regarding educational research 
and then address the problems of organizing research, development, 
diffusion, and evaluation so that they become more potent forces In 
shaping the nature of schooling. 



Federal Reoearch and Development Policy 



Attempting to identify trends in federal policy regarding educational 
research and development may be premature , particularly since for the 
next few years that policy is likely to originate in the terra incognita 
of the National Institute of Education* 

The federal government is currently the single largest source of 
funds for educational research and development* The policy of the 
disbursing agencies appears to be that educational research (both basic 
and applied) and development can best be pursued within universities 
(including R&D centers) and other public, nonprofit agencies (e.g*, 
regional laboratories). Two cautions should be raised concerning such 
policy: (^) it runs counter to the organization of applied research and 

development in the nation; and, (^ it threatens to subvert the unique 
contribution which the universities can make to the production and 
distribution of knowledge through basic research* 

Although in recent years the majority of funds for research and 
development in all fields has been disbursed by the federal government, 
the universities have been the largest user of funds for basic research, 
and industry has been the largest user of funds for applied research 
and development (see Tables 1 and 2). For example, in 1968-69, 6A% 
of all funds spent on applied research and development were spent 
by private Industry. Only 9% of the funds for applied research and 
development were spent in universities. At the same time, most applied 
research and development in education was being conducted in universities. 
If applied research and development in other fields are conducted 
largely within private industry, why should applied research and devel- 
opment in education be conducted in universities? Whether one regards 
current allocation patterns with shock or with approbation ultimately 
goes to the heart of his social-political philosophy. The conservative 
frame of mind pauses to search out the wisdom — if there is any — in the 
established order. Policy-makers should do no less in thinking through 
the problems of organizing educational research and development. 

There appear to be few reasons to doubt that basic research will 
continue to emanate primarily from the universities. These institutions 
are designed for the discovery of new knowledge and the training of the 
next generation of discoverers. A case can also be made for applied 
research being conducted in universities, although much of it might 
quite appropriately be left to industry. On balance, the success of 
universities in development work is more equivocal* In some cases (e.g., 
government sponsored work for the military) development has been disruptive 
of the business of the universities. Scholars have been removed from 
their research pursuits in many instances and their services as teachers 
of both undergraduate and graduate students have been lost, far more so 
at the undergraduate level. 



Research 




Greater payoff can be realized from educational research to the 
extent that: (a) it adopts more of the perspectives afforded by all 

of the social sciences (not just psychology); {h) its relationship to 
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practical problems of development is examined critically; and, (£) it 
is given financial support. Each of these considerations is discussed 
briefly below: 

Interdisciplinary research . Making educational research multi- 
disciplinary is the new shibboleth of the research community. The 
slogan is »roper and we support it with few reservations. The problem 
and prospe :to have been discussed in detail elsewhere (Glass, 1968) 
and, in the interest of brevity, will not be repeated here. 

R elationship £f research to development . In an earlier section 
of this paper, a view was presented of the relationship of research 
and development in which development took its direction from basic 
research. (The view could be characterized as ; he **deve]opment-catches- 
crumbs-from-the-table-of-research** model.) Other conceptualizations 
of the relationship of research to development are possible and, to 
seme extent, defensible. Scrlven (1969, pp. 18-21) argued that: 

”It is . . . much better to move into theory 
exactly where and only when obliged to by the 
combination of data and needs that define our 
task. Speculation in the absence of a clar- 
ification of these parameters is too often merely 
idle, the kind of irresponsible gambling with 
'Society's resources that is lauded in cheap 
histories of science and was once a rich amateur's 
prerogative** [p. 19] . 

". . . the selection of problems for research 
should be much more closely guided by consider- 
ations of practical needs and advantages" [p. 20]. 

Scriven's remarks were made in the context of a discussion of pure 
and applied research in psychology. 

Space does not permit our going deeply into the problem of the 
"power structure" among research, development, and evaluation. This 
power structure, with its order of dominance (who tells whom what to 
do?), is no less important than the professional reward structure dis- 
cussed later in this paper. 

The view suggested here of development leading research is at odds 
with the earlier view of research leading development. The two views 
are obviously contradictory, and we are Incapable of resolving the 
contradiction. Indeed, it seems only honest and sensible to acknowledge 
the uncertainty that causes us to present the two contradictory views 
without pretending to know to which one we should subscribe. ("I contradict 
myself? Very well, I contradict myself." — Walt Whitman) 

Financial support . Finally, more money spent on educational research 
and the training of researchers will result in more research and more 
knowledge. Such a commonplace would hardly seem necessary to state 
were it not that there are signs that allocation policies are being 
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framed and implemented in the belief that decreased support for educa- 
tional research will cause neither a decrease in the number of persons 
trained to do research nor in the knowledge they discover. 



Kvaluation 



The single most important impediment to the ef f ective utilization 
of educarional evaluation talent and expertise is the reward structure 
within universities. The scholarly activities encouraged and rewarded 
by the university promotion policies are more research than evaluation. 
Few persons have the professional security to buck the system and en- 
danger their futures in the academic world . 

The power of the professional reward structure to enforce conformity 
to traditional research was observed at a recent meeting of the Society 
for Applied Anthropology. At a session on anthropological approaches 
to educational development and evaluation, three anthropologists re- 
counted their genuinely exciting experiences in building and assessing 
special educational programs for various American Indian tribes. During 
an open discussion following the presentations, a senior member of the 
discipline rose to reaffirm that "anthropology is the disinterested 
stu y of mankind and that his colleagues' work saddened him because 
t represented the prostitution of a noble science. The threat in his 
remarks was evident even to an outsider. He was calling his errant 
colleagues to return, and if they did not, they would lose their identity 
as anthropologists. The accused attempted to defend themselves, but 
inappropriately. They sought to justify their involvement in applied 
problems in education on the grounds that their experiences were making 
them more knowledgeable anthropologists and that basic knowledge about 
antnropology would be the ultimate result of their labors. To the 
outsider, they appeared to give the wrong reason for doing the right 
thing. First, development and evaluation will probably contribute very 
little to basic knowledge in any of the social sciences; unfulfilled 
expectations and a reaction against development and evaluation are the 
likely consequences of justifying evaluation on its "fall-out" for 
basic research. Evaluators must be careful not to promise too much, 
particularly as regards the basic knowledge likely to result from their 
efforts. Secondly, that billions of dollars are spent yearly on educa- 
tion and that human welfare has some relationship to the activity are 
reasons enough that more than a handful of social scientists should 
become involved in development and evaluation. If basic anthropology 
never gained one iota of knowledge from the efforts of 10% of all 
anthropologists working in behalf of educational Improvement, society 
would probably be none the worse off. 



The traditional reward structure in the university and in a field 
of scholarship is based ostensibly on teaching (which is usually assessed 
on tne basis of heresay evidence from students), service (which is seldom 
'HI lie and thus seldom assessed at all) , and research (which — in spite 
of our protestations to the contrary— can be measured as validly as 
IS currently the practice by weighing a man's publications). Promotion 
within the university and prestige within the discipline depend almost 
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entirely on a man's published works. (We are not prepared to take the 
final step in cynicism and argue that the length of a scientist's 
publication list is uncorrelated with his contribution to the discovery 
of knowledge. Science is public — a ’’superculture” that spans the 
globe, in Boulding's words — and publication is essential to its growth.) 

As was argued earlier in this paper, the results of an evaluation 
are of severel/ limited generality; the audience for the report is 
rarely very large. The consumers of an evaluation are seldom capable 
of appreciating the expertise which the evaluator lavishes on the project. 
One may show off to one's colleagues, but one must be humble and un- 
complicated with one's clients. Thus evaluation does not offer the 
opportunities for publication that the researcher enjoys. As long as 
promotion and prestige are tied to publication, the professional reward 
structure will militate against involvement in evaluation in favor of 
traditional patterns of research. 

There is probably a greater chance of successfully creating new 
institutions which are alternatives to the universities and which have 
different reward structures than of altering the existing reward struc- 
tures of the universities and scientific disciplines. (There are good 
reasons for not tampering with the organization of the universities to 
give greater emphasis to evaluation. These reasons will be examined 
in a later section on the implementation of development where they apply 
with more force.) The RAND Corporation is the preeminent example of 
an institution which has sustained research of an applied sort, much 
of it classified and thus unpublishable. Such institutions can be 
successful in rewarding accomplishments of a type that go unrecognized 
in the larger profession. Such alternative institutions perform a 
function far more significant than is generally credited them. They 
represent the most promising mechanism for bringing evaluation into 
education. 



Development and Diffusion 



These two inquiry-related activities will be dealt with together 
because we shall propose a common solution to the problem of increasing 
their impact on education. 

As was Implied earlier, the growth of writings on educational 
diffusion ("change agents," "linkage agents," etc.) has been prolifer- 
ative and is in need of a loyal opposition. Nearly all thinking about 
diffusion begins with the same premise: a new role (institution or 

organization) must be created to effect change in the schools. As 
Rankin and Blanke (1968) noted: 

"Two assumptions re-occur frequently in the 
literature on educational change: (1) there 

is a large gap between theory and practice, 
and (2) special organizations must be created 
and individuals trained to bridge this gap 
if educational Improvement is to be consis- 
tent, effective, and efficient" [p. 1]. 
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To our knowledge, alternative assumptions have not been examined 
seriously; all thinking has been directed toward delineating the 
functions of this new role, the disseminator. In our opinion, the 
major impediment to change in the schools is probably not the absence 
of an agent to diffuse educational developments and thereby link the 
the classroom to the R&D center. 

Most writing in the educational literature about diffusers is 
based more or less on the "county-agent” model from agriculture. 

However, schools are not farms (as is obvious), and the important 
respect in which they differ is the nub of the problem of getting the 
former to change. The farmer participates in an exchange economy and 
much of his behavior is under the control of the market. The county 
agent has been instrumental in improving the productivity of agriculture, 
but his role tends to be overemphasized by educationists who are also 
prone to minimize the roles played by farm implement salesmen, the 
motivating effects of the marketplace itself, and even the negative 
effects of "cooperatives” and subsidy programs. The schoolman works 
in what has been called a "grants economy,” and his behavior is accord- 
ingly quite different from the farmer’s. 

Boulding (1970) analyzed the predominant economic relationships 
among men and organizations and the implications of these relation- 
ships for behavior. The traditional market is based on an exchange 
relationship ; A gives something to B, and B gives something of equal 
value to A. Feedback on whether the exchange has produced a net gain 
for both A and B is immediate. Allowed to function long enough, the 
market based on this simple exchange relationship insures the continued 
welfare of both A and B. The "market economy" is organized around 
this simple act of exchanging. A second important relationship is 
integrative (Boulding, 1970). In an integrative relationship A gives 
somethi*ig to B because he likes him, is concerned about B’s welfare, 
because posterity will benefit, etc. A does not expect feedback from 
B; indeed to expect it is at cross-purposes to the altruistic nature 
of the relationship. The integrative relationship is the basis for 
what Boulding has called the "grants economy." The grants economy 
is that sector of the economy which involves the granting of money from 
one person (or organization) to another. The bulk of the grants economy 
is in the public sector though private philanthropy is by no means 
negligible. In large part, governments act as the granters, spending 
tax dollars on public works such as welfare, defense, police protection, 
and schooling . 

The grants economy, which is estimated to have risen from about 
3% of the Gross National Product (GNP) in 1900 to approximately 13% 
of the GNP today (Boulding, 1968), will face hard times over the next 
few decades: 



"We now seem to have reached the point where strong 
tax resistance has developed in American society, 
which is a symptom of the fact that the public 
grants economy has been stretched to the point 
where the sense of community, both local and national, 
will not support much expansion of it. We see this 




in the failure of miliage increases, and school bonds 
in local elections . . . Under these circumstances, 
the assumption that the grants sector of the economy 
can be indefinitely expanded is quite unrealistic. 

This fact is of crucial importance to the educational 
industry, which depends so much on the grants sector 
of the economy. We may be facing a real disaster if 
the expansion of the educational industry, which is 
so clearly required by the dynamics of the society, 
cannot be financed because of the limits on the 
grants economy" [p. 369]. 

The percentage of the GNP expended on public elementary and secondary 
education in the United States is depicted in Figure 6. The percentage 
of the GNP involved in education (public and private) at all levels has 
grown from 3% in 1900 to 7% in 1970. It is clear that education is 
becoming increasingly expensive as are all areas of the grants economy 
in this century. As in the case of local police (which, heedless of the 
bumper stickers, the public is not supporting financially), the ultimate 
result of increasing ex^^enditures of public dollars is a clamping down on 
grants. Education may face this prospect for several years to come. This 
possibility should make planners leary of creating new roles and staffing 
them at public expense, particularly if the roles are nebulous and their 
value still to be demonstrated. <^^ne disseminator or change agent fov each 
public school district in the nation would Increase expenditures for 
schooling by 1% (20,000 disseminators X $15,000/dissemlnator = $0.3 billion, 
which was 1% of the $31.5 billion expenditure for public elennntary and 
secondary education in 1967). Such proposals must be scrutinized care- 
fully in an age when all manner of educational reformers can be found 
who wish to enrich education at public expense. 

In this context, a relevant question is whether many development 
and diffusion activities can be performed efficiently by educationists 
in any event. Bureaucracies are notoriously ineffective in producing 
or nurturing change (Thompson, 1965; Merton^ 1950) and most educational 
institutions exhibit many characteristics of classic definitions of 
bureaucracy. In other highly bureaucratized sectors of the grants economy 
an incapacity to engage extensively in development and diffusion activities 
is apparent. Witness the military’s subcontracting of virtually all work 
on developing transportation and weapons systems to private industries 
such as Boeing, Dow Chemical, and Sperry-Rand. In education, this in- 
capacity has been denied and one finds many proponents of the "I’d rather 
do it myself" approach. Taking this position in the absence of any 
evidence that educationists either can or should assume major responsi- 
bility for educational development and diffusion seems to us to be ill- 
advised. 

The net effect on educational practice of all the scholarly papers 
and conferences on educational change which could be produced , all of 
the laws Congress could pass , and all of the federal programs which could 
be mounted would be insignificant beside the Impetus for change that 
would result from moving schooling out of the gran ts economy and under 
the influence of the market mechanism . Schooling and most educational 
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development and diffusion activities should be turned over to the private 
sector of the economy. There appears to be shockingly little faith in 
capitalism among the same schoolmen who regard the subject as a corner- 
stone of a good portion of the public-school civics and history curriculum. 
The knee-jerk bureaucratic response of passing a law or "mounting" a 
government program whenever a problem is encountered is wasteful and a 
potential failure as a means of promoting educational change. Govern- 
ment at all levels would be well advised to leave educational develop- 
ment and diffusion to private enterprise. That they intend not to is 
testimony to the singularly liberal failing of a lack of faith and 
patience in the market mechanism. In fact, the Involvement of govern- 
ments at all levels in schooling is one of the greatest impediments to 
the proper functioning of the market mechanism. In our opinion, moving 
schooling out of the grants economy into a market economy would cause 
educational development and diffusion to flourish. 

We recognize that our recommendation has utopian overtones. However, 
the alternative of leaving educational development and diffusion largely 
to private industry must be given serious consideration. If no other 
reasons existed, the hard times facing the grants economy are reason 
enough not to place additional economic strain on public support for 
education until and unless It can be demonstrated that there Is no 
alternative way to bring about educational improvement. 

There are some development and diffusion activities for which 
educationists might properly assume responsibility; but, these represent 
a relatively small portion of the whole spectrum of diffusion and develop- 
ment functions. For example, the dissemination of scientific knowledge 
in education might appropriately be considered an inquiry-related activity 
to which educationists should give attention. Encouraging utilization 
of scientific information is a fragile undertaking that might be destroyed 
by the imposition of charges for information that would be necessary if 
the endeavor were relegated to profit-making organizations. Costs for 
information might discourage the use of information sources, however 
efficient. 20 Dissemination of scientific knowledge should be nurtured. 



^We Include under the rubric "scientific knowledge in education" 
not only results of basic research, but also products of development 
that meet two criteria: (^) they have been thoroughly tested through 

evaluation and applied research studies and found to be of worth, and 
(^) they cannot readily be used for profit-making by private industry. 
For example, if a particular pattern of staff utilization had been 
developed and shown through applied research to produce salutary effects 
on student achievement, it might well fit here since a plan for staff 
utilization Is hardly a commodity that could be copyrighted or marketed 
by private Industry. Thus, dissemination of information about this type 
of "product" would be the responsibility of "educational disseminators." 

20 

^The chesis is developed elsewhere (Glass » 1969) that educational 
information is a highly elastic commodity — in the economic sense; 
when its cost in time and inconvenience rises above a minimal level, 
the schoolman gladly substitutes folklore and tradition for it. Also 
see Stake (1967b) from whom these ideas were taken. 
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at public expense. It is here that we see further thought about dif- 
fusion profitable for the field of education. 

Diffusion of developed products that are marketable (e.g., curriculum 
materials, audio-visual devices, computer programs, technological 
advances) demands far more personnel and financial resources for dis- 
semination, demonstration, and assistance with adoption than does the 
simple dissemination of knowledge discussed above. It is this former 
type of diffusion — the major portion of the whole — that should be 
moved from the grants economy to the private sector. 

There is evidence that much of the talent for developing educational 
materials, especially curriculum materials, exists in universities, 
regional laboratories, and other educational settings. Productive efforts 
of many curriculum development projects (e.g., PSSC, BSCS) are well known 
and it is questionable whether private industry would expend the large 
sums necessax^y to develop similar materials that might be viewed as "high 
risk" in the profit-making sense. Certainly some public subsidies for 
such high-risk development ventures will be necessary, but they will 
demand considerably less resources and energy than the development of 
most educational hardware and software. By excluding this type of 
development which private enterprise is equipped and willing to assume, 
the burden on public expenses for schooling could be reduced and efficiency 
could be increased. 



Trends Toward the Private Sector 



The motivation for developing and disseminating educational pro- 
ducts must originate with schoolmen. The motivation is lacking as long 
as schoolmen are spared the obligation to maintain a market relationship 
with the public. Any idea about education in the 1970 *s worth proposing 
has had previous advocates. We have no priority claim on the concept 
of an open market in schooling. Friedman (1970) adumbrated a plan for 
instituting the market mechanism in higher education. Boulding (1968) 
advocated more moderate refom of higher education than Friedman, though 
his intention was similarly to move schooling more toward an open market. 
Levin (1968) examined arguments on both sides of the open-market question 
and called for experimentation to resolve the uncertainties in the 
various proposals. (Also see Carr and Hayward, 1970.) 

In the fall of 1972, the Office of Economic Opportunity will 
instigate an experimental "voucher plan" in one or two communities. 

Parents will be offered vouchers equal to the per-pupil expenditure 
for schooling in their community; the vouchers can be used to pay for 
the education of their child in any school in the community: public, 

private, or parochial. Another hopeful sign of change, which will have 
to proceed cautiously to avoid being destroyed by overstimulated expec- 
tations and unsubstantiated claims, is contract schooling or "selling 
grade levels." The initiative in this area has been taken by private 
firms such as the Educational Developmental Laboratories of Long Island, 

New York, and the Behavioral Research Laboratories of Palo Alto, California. 
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One of the more propitious occurrences In educational development 
of the last fev years is the recent decision of the U. S. Office of 
Education to reverse the policy of public domain rights on educational 
developments resulting from government supported research and develop- 
ment. The current plan to permit USOE sponsored research and develop- 
ment agencies to receive one-half of all royalties from the marketing 
of their development promises to stimulate a great deal of worthwhile 
effort. The biggest mistake which the R&D agencies could make would 
be to plow all of the profits back into the enterprise and thus weaken 
the profit motive in individual workers. 

Computer-assisted instruction is currently an underdeveloped area, 
at least the software side. The dearth of instructional materials for 
CAI could be related to problems of authors’ achieving wide dissemination 
of their product while protecting copyrights. The most hopeful proposal 
for stimulating CAI development is large time-sharing computer systems 
among universities in which authors are paid royalties for the use of 
their instructional programs much as actors are given *'resid\ials" for 
repeats of their television performances. 



Problems with the Market Mechanism In Education 



The market mechanism works so poorly in education today because 
schoolmen do not participate in a market relationship. Funds are granted 
to them, and they are not required to present feedback to the grantor 
(in this case, the public) as evidence of how well the funds are being 
spent. No genuine exchange relationship can be established where one 
party is not accountable to the other. The first step toward instituting 
the market mechanism in education is to complete the feedback loop from 
the school to the public, to make the schoolman accountable for his 
successes and failures. To base education on an exchange relationship 
between the public and the schools would create a high demand for educa- 
tional development, evaluation, and diffusion. Much of the literature 
and planning for diffusion proceeds from the premise that the lack of 
progress is attributable to deficiencies in the supply side of the 
market. However, the lack of progress in educational diffusion is a 
result of a nonexistent demand. 

The sluggishness of the market for educational development is often 
attributed to the suppliers instead of those who control demand. In a 
study of the educational products industries by the Institute for 
Educational Development, the following conclusion was reached (quoted 
from Educational Research and Development In the United States , prepared 
by the Bureau of Research, U.S. Office of Education, July 1969): 

"lED (Institute for Educational Development) found 
that, with very few exceptions, company officials’ 
perceptions of their role in the educational enter- 
prise were quite limited and relatively passive. 
lED concluded that restricted and passive role 
perceptions appeared to limit the possibilities for 
research and development activities within the 
educational materials industries" [p . 250] . 
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To our knowledge, no one has ever laid the blame for the disappear- 
ance of the kerosene lamp industry on the passivity or limited role 
perceptions of its officials. 

One of the more significant phenomena with which textbook publishers 
mst deal is the fact that approximately 13% of the market (representing 
the populations of Texas and California and not counting many large-city 
school districts) is completely closed; textbook adoptions in Texas and 
California are statewide. Although this may appear to create lively 
competition among publishers, the effecti\^e result of these markets 
being closed is to discourage innovation in the educational products 
industry. Publishers assess each new proposal for a textbook in terms 
of whether it will sell in Texas or California. 

The difficulties with the proposal to base schooling on the market 
mechanism are the soft spots in every purely capitalistic system, namely 
the desire of the top dogs to eliminate competition t.irough monopolies, 
trusts, cartels, etc., and to sti le feedback (criticism) concerning 
the consumer's satisfaction with the exchange. Instances of the former 
strategy abound in the history of this country. The corruption of the 
market mechanism is illustrated by such instances as the Consumers Union 
finding it Impossible to purchase advertising space in leading newspapers 
for 15 years, the intrigue of Ralph Nader vs. General Motors, and the 
birth trauma of the Educational Products Information Exchange. 21 

Perhaps the open market in schooling could never remain uncorrupted 
by the educators' desire to eliminate competition. The Toronto, Ontario 
school system is believed by some outsiders to be an open market. It 
is alleged that parents may send their children to any school in the 
city, public or private, and that each school receives financial reim- 
bursement from the city based on its enrollment. It is further claimed 
— by opponents of proposals recounted here — that this "open market" 
has resulted in a school system which is as traditional as any comparable 
system in the U. S. However, to the insider, it is apparent that the 
school officials in Toronto have so successfully deemphasized and 
tangled in bureaucracy the parents* right to freely choose schools that 
neighborhood schooling prevails almost without exception. 

There are many reasons why a pure laissez-faire market in schooling 
might not be practical. The public could not tolerate periods of suspended 
services when one "firm" goes out ^ business. The long-run interests 
of all are served by educating all children, whether they can afford it 
or not. And so forth. There are many compromises with the ideal of an 
open market that could be instituted in education to protect the common- 
wealth. Loan programs administered through state or federal governments 
(or even through private enterprise) could be expanded at the college 



21 

EPIE was reported to have been threatened with an Injunction 
by audio-visual aids manufacturers in connection with a simple descriptive 
report on the features of the overhead projectors on the market (see 
School and University magazine, October 1967). 
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level and — come the millennium — extended down to the secondary and 

7 9 

elementary school level. 

Similar considerations apply to problems of moving educational 
development and diffusion activities Into the open market. There are 
several developmental activities that probably would not be conducted 
well by private industries. For example, in the drug industry, 
pharmaceutical research In the U. S. led to ouch advances In drug 
development that the European pharmaceutical market adopted U. S. develop- 
ments almost directly. When the U. S. government placed ceilings on 
profits In drug manufacturing, the drug Industry responded by curtailing 
research and development efforts, eventually moving full cycle and be- 
coming dependent on developments In the European market. The results — 
Including problems with Thalidomide and other drugs developed In other 
countries — were deleterious In many Instances. The problem Is that 
genuine altruism Is rare, and research and development for the good of 
mankind Is given short shrift when the margin of profit falls below a 
level acceptable to private enterprise. It Is because of this problem 
that It was suggested earlier that certain development and diffusion 
functions must be retained by educationists. 



Consumer Guidance 



In one form or another, the Establishment response — the establish- 
ment of researchers, evaluators, and developers as well as schoolmen — 
to the proposal of market schooling Is that the ''parents do not know 
what's good for their children." As regards schooling the public can- 
not be trusted to act In their own best Interest, the Establishment 
maintains. The argument that the educationist subculture harbors a 
set of values and goals for schooling which Is manifestly superior to 
those of the nonprofessional community Is singularly unconvincing. The 
current fashion In urban and compensatory education — community parti- 
cipation — Is based on the antithetical premise that it Is often better 
to let people make their own mistakes, even at the expense of their short- 
run gain. (The elitist Is unwilling to forfeit the short-run gain, agreeing 
with Keynes that we'll all be dead In the long run.) The Institution of 
the market mechanism in schooling would restore a precious freedom to the 
public, the right to be wrong. One of many plausible explanations (and 
quite likely a contributory cause) of the deplorable state of much of the 
public school system Is that elitist educationists have so thoroughly 
protected the public from erring on Its own behalf that they produced 
feelings of Impotence and Indifference. The freedom to choose necessarily 
entails the freedom to choose unwisely. But those very freedoms are 
vital If each man Is to come to believe that his life counts for some- 
thing In a society that Is Increasingly managed and programmed by an 
unresponsive mandarin class. 



^^Professor Killingsworth of Michigan State University has proposed 
a system of federal banks which would loan money to any Individual 
desiring a college education; the loan would be repaid through a surcharge 
on the Individual's Income tax. 
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At another level, the Establishment of researchers, evaluators, 
and developers might express similar concerns about whether or not 
schoolmen "know what is good for the children." Such a concern need 
not be Interpreted as critical of schoolmen in particular but rather of 
the gullibility of consumers in general. The correlation between the 
volume and quality of advertising and the volume of sales (regardless 
of quality) Is substantial enough to give pause to the thoughtful. The 
dollars expended on hair restoratives and glyphs are cases in point. 
Consumers are frighteningly susceptible to purchasing for many reasons 
unassociated with the real worth of the product; educators are not a 
breed apart. 

Despite the fact that both parents and educators are likely on 
occasion to be mislead by hucksters, charlatans, or Madison Avenue, we 
do not endorse the position that either educators or the public must 
be "protected" against making wrong choices. Instead, we are optimistic 
that educational consumers can be educated to consume intelligently; 
the difficulty of choosing wisely should not be overemphasized. 

In open market place, the public could be aided by agencies analogous 
to the recently formed, federally supported Consumer Guidance Commission. 
This commission was formed to identify and rate unreasonably hazardous 
household products. Products are not endorsed, but lists of those that 
should be modified or removed from the market are distributed. A similar 
strategy could be employed in education as a public service and, appro- 
priately, at public expense. In this way, government could have a 
significant positive impact on the working of the market, thus producing 
salutary effects on the development and diffusion of educational products. 
The formation of EPIE was a step in the right direction; more vigorous 
steps will be needed. For example, agencies might be established to 
evaluate not only educational products but also assess the effectiveness 
of existing or newly proposed educational processes (e.g., organizational 
or staffing plans, and pedagogical techniques). Such agencies might, 
through systematic evaluative inquiry, make recommendations about the 
relative effectiveness of alternative educational processes under several 
types of conditions. Such recommendations, coupled with information 
generated by product evaluating agencies such as EPIE, would provide 
considerable help to schoolmen and the public in choosing wisely among 
the variety of processes and products they might use to educate their 
children. 

It would be foolhardy to underestimate the practical problems of 
shifting schooling even a small distance on the grants-exchange economy 
continuum. Obviously the task falls to the experts in economics, school 
finance, and related fields. We wish only to support the notion in 
principle that the development and dissemination of educational products 
will proceed more rapidly and to greater effect if carried out in the 
private sector than If conducted In the network of universities and non- 
profit agencies established by current allocation policies. 
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Conclusion 



Perhaps Che best conclusion we could give Is to state as succinctly 
as possible Che general concerns with educational inquiry which formed 
the background for this paper: 

1. There are probably more slxallarltlcs than differences between 
research and evaluation. However, the similarities are generally well 
known and there Is a danger that overemphasizing them at this time does 
a disservice to both researchers and evaluators. More is to be gained 
now rom drawing distinctions among inquiry and inquiry-related activities 
than from emphasizing chelr commonality. 

Our concern with definitions Is not mere scholasticism, but rather 
an acknowledgement chat the conceptions of types of inquiry guide behavior 
and determine to some extent the nature and success of these activities. 

2. We cannot be confident of any recommendations about how research, 
evaluation, and development should be Interrelated. We hope not to appear 
CO be credulously accepting of the claim chat development must follow 
basic and applied research; at the same time we reall<.c the potential 
hazards of turning Che relationship around so chat basic researchers 

take chelr direction from evaluators and developers. 

3. Support for basic research in education is threatened by a 

lack of faith in the long-range payoff of inquiries motivated by curiosity. 

4. The full utilization of educational evaluation is impeded by 
the professional reward structure within educational research. 

5. Development of educational products and their dissemination are 
Impeded by the economic structure of the schools and not by a slothful 
and passive educational products Industry. These activities should be 
conducted primarily in the private sector of the economy and not pursued 
by universities. 
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A CRITIQUE OF THE PAPER 



Paul U. Hood 

Far West Laboratory for 
Educational Research and Development 



Two Perspectives; Inquiry and Practice 

^^Educational Inquiry and the Practice of Education'* is in fact two 
esf^ays. One deals with disciplined inquiry, the other with accountable 
practice. In the first essay the authors examine, from the academic 
perspective of "disciplined inquiry," the nature and relationships of 
four processes: research, evaluation, development, and diffusion. A 

content analysis shows that the major part of this essay deals with 
distinguishing "basic" research from "value oriented" outcome evaluation, 
both of which are depicted as falling entirely within the area of 
disciplined inquiry but differing in many characteristics. By contrast, 
less than one-fourth of the first essay deals with development, diffusion, 
or the relations of development and diffusion to each other, to research, 
or to evaluation. This highly disproportionate treatment among the 
four subject areas is possibly understood by examining their Figure 3, 
depicting the inquiry domain. Here, educational diffusion lies totally 
outside the area of disciplined inquiry and educational development is 
only partly within. Moreover, both diffusion and development are depicted 
as relatively small areas when compared to either educational research 
or educational evaluation. This apparently is the perspective of Glass 
and Wo r then. 

By contrast, if the perspective of educational change and improve- 
ment of practice is taken (instead of "language usage". . .), we might 
arrive at a substantially different view as in Figure 1. 

In this figure, educational diffusion, development, evaluation, and 
applied research are all related to one another and are almost totally 
within the domain of concern for improvement of educational practice. 

On the other hand, "basic" research is a peripheral activity, relating to 
educational practice improvement primarily in terms of the relationship of 
basic research to applied educational research and to educational evaluation. 

In view of the title of the Glass and Worthen paper, it is unfortunate 
that their diagrams regarding disciplined inquiry were not extended to 
include explicitly the domain of educational practice. 



The Place of Applied Research 

The reader may note that in Figure 1 basic and applied research 
are treated as separate but overlapping activities. Although Glass 
and Worthen have not entirely ignored applied research (see pp. 73-75; 
99-101), they have given most of their attention to basic research. By 




FIG. 1. The inquiry domain and educational practice. 



opposing basic research to one kind of evaluation, ^ the authors make 
a number of useful distinctions between these two rather well removed 
activities. But in so doing they have managed to skirt some Important 
and conceptually more difficult areas. It is the nature of applied 
research, knowledge based development and diffusion, and evaluation, 
and their relationships that seem to pose our current conceptual 
problems, at least with respect to the Improvement of educational practice. 

Consider, for Instance, the 10 characteristics of inquiry that 
the authors propose as a basis for distinguishing research from evaluation. 
Not all of these distinctions remain valid if we attempt to distinguish 
applied (rather than basic) research from evaluation, viz: 

1. Motivation of the inquirer. . . evaluation 

is Intended to contribute to the solution of a 
particular kind of practical problem" [p. 80]. 

2. Autonomy of the inquiry. "... evaluation 
is undertaken at the behest of a client, but the 
researcher sets his own problems" [p. 83]. 




^"Unless inclusion of hybrid inquiry activities becomes essential 
to the point under consideration, the terms 'research* and 'evaluation* 
will be used ... to refer to the 'purest' form of each, basic research 
and outcome evaluation" [p. 77]. 
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7. "Universality" of the phenomena studied. "Program 
evaluation is depicted as concerned with a phenomenon 
(an 'educational program') which has limited general- 
izability across time and geography . . . the concepts 
upon which educational research is carried out are 
supposed to be relatively permanent, applicable to 
schooling ne€u:ly everywhere, and they should subsume 
a large number of Instances of teaching and learning" 
Ip. 86]. 

With respect to these characteristics It seems that applied research 

is more like evaluation than basic research. 



Decision-oriented or Value-oriented Inquiry 

Despite these possible flaws, the first essay is quite useful. Their 
definitions of research, evaluation, development, and diffusion (pp. 69-70) 
do provide a useful point of departure, and viewing these activities 
from the perspective of disciplined inquiry may be enlightening. Per- 
sonally, I have found Cronbach and Suppes' treatment of disciplined inquiry 
In education to be quite helpful, especially in its distinction of "con- 
clusion-oriented" and "decision-oriented" inquiry. Glass and Worthen 
apparently relate these two concepts respectively to research and eval- 
uation, viz.. 



"Conclusion-oriented inquiry is much like 
what is here referred to as research; decision- 
oriented Inquiry typifies evaluation as well as 
any three words can" [p. 82]. 

My reading of Cronbach and Suppes leads to the impression that 
decision-oriented inquiry is a broader concept which certainly Includes 
parts of applied research (as well as operations analysis and operations 
research) in addition to evaluation.^ 

Although this equation of decision-oriented inquiry with evaluation 
suggests a very broad interpretation of evaluation. Glass and Worthen 
are, in fact, much more restrictive. They would explicitly exclude 
"program monitoring" or "process evaluation" as well as "context eval- 
uation," and they also reject "decision-centered" evaluation models; 
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Glass and Worthen note: "Two activities that might be considered 

variants of applied research are 'institutional research* and 'operations 
research, ' activities directed toward supplying institutions or social 
systems with data relevant to their operations. To the extent that the 
conclusions of inquiries of this type are generallzable, at least across 
time, these activities may appropriately be subsumed under the research 
rubric. However, where the object of the search becomes nongenerallzable 
information on performance characteristics of a specific program or process, 
the label "evaluation" is more appropriate [p. 75]. 



arguing that these models "neglect two fundamental points of Scrlven's 
definition of evaluation, viz., that the 'activity consists in the . . . 
combining of performance data with a weighted set of goal scales to yield 
either comparative or numerical ratings, and in the justification of 
(a) the data-gatherlng instruments, (^) the weightings, and (O the 
selection of the goals'" (Scriven, 1969, p. 40)[p.76]« 

Their arguments against a decision-oriented approach seem labored, 
for on the one hand they assert: 

"Decision-centered evaluation methodologists argue 
that values are included in their . . . models 
because a decision is always the revelation of a 
value" [p. 76], but they also assert "For a values- 
centered evaluator, however, decisions are implicit 
in evaluation, l.e., in the process of measurement 
against value scales, integration of measures into 
value-statements, and the Justification of the 
measurement and the means of integrating the measure- 
ments • • • However, a decision centered evaluation 
model can be applied without concentrating attention 
on the process by which a decision-maker integrates 
information into an overall judgment [pp. 76-77]. 

In further rebuttal, they note: 

"It would be satisfactory to disregard the direct 
assessment of value and merely provide data to 
decision-makers if decision-makers ' preferences 
were always logical, rational, intelligent revela- 
tions of value" [p. 77 ]. 

Apparently, it is their distrust of the declslon-nakers ' logic, 
and their trust of the value judgment of the professional evaluator which 
partly explains their preference for the value-centered rather than decision- 
centered approach. The practitioner, on the other hand, may distrust the 
value judgments of the professional and will usually depend on his own 
judgment to integrate information in any truly crucial decision.^ 

Glass and Worthen note that: "Evaluation can play many roles . . . 

However, the go^d of evaluation must always be to provide an answer to 
an all-important question: Does the progran wider observation have 

greater value than its competitors or sufficient value of itself that 
it should be maintained?" [p. 77]. But whose value judgments are really 
brought to bear on this all-important question? Decision-oriented models, 
at their best, do grapple with what kinds of information are needed by 
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See Howard Raiffa (Decision Analysis , 1968) for a discussion when 
business executives will rely on technical evaluations of investment 
alternatives* 



and can be used by the decision-maker (s) . ^ The risk of value-oriented 
models is that they may reflect too greatly the biases of the profes- 
sional evaluator and cope only with technically tractable value scales 
and measures, sometimes ignoring information and values of substantial 
relevance but beyond the practitioner's capacity to analyze or communicate. 

Given that one fully realizes that Glass and Worthen are concerned 
primarily with basic research and value-oriented outcome evaluation, 
then their extensive treatment of the characteristics that distinguish 
these two activities is an important contribution. I found the sections 
on properties of the phenomena which are assessed (pp. 83-84 )> the 
"universality" of the phenomena studied (pp. 84,86), and investigative 
techniques (pp. 86-89) to be of particular interest. And I noted with 
approbation their statement: "While there may be legitimate differences 

between research and evaluation methods, we see far more similarities 
than differences between research and evaluation with regard to the 
techniques by which empirical evidence is collected and judged to be 
sound" [pp. 89-90]. 



Implementation of Research and Evaluation 

The same problem with "pure" definitions of research and evaluation, 
ignoring applied research, carries over into their discussion of the 
Implementation of research and evaluation; for as they note: "The mean- 

ings which 'research,' 'evaluation,' .'development' and 'diffusion' hold 
for a person make practical differences in how he will pursue inquiry" 

[p. 91]. With respect to disciplinary base, they observe that the ed- 
ucational researcher can afford to pursue inquiry within one paradigm 
and the evaluation team cannot. I would observe that in practice most 
applied research, like evaluation, profits from a multidisciplinary 
base. Similar comments can be made with regard their contrasts of five 
elements of graduate level training programs for researchers and eval- 
uators. The applied researcher, unlike the basic researcher, does 
profit from exposure to more than one discipline, from acquaintance with 
a breadth of techniques, from more organizational and management train- 
ing, and from exposure to the field during internship training. 

This narrow view of research and evaluation must also be considered 
when one encounters the beginning of their summary of the relationship 
between educational research and evaluation: "Evaluation borrows inquiry 

techniques and a knowledge base for recognizing value from research and 
contributes little in return" [p. 95]. Evaluation In practice (as opposed 
to an academic position as to what it should be) actually borrows most 
of its techniques from applied research and development, and it derives 
its value systems directly from practice rather than basic research. 
Educational research has all too often succeeded in little more than 



Coping with the integration of multidimensional value systems and 
making choices among them is a major challenge that has been dealt with 
explicitly by systems and operations analysis. (e.g. , Hanssman, 1968; 
Muller and Starr, 1960; Churchman, 1961; English, 1968). 
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confirming, albeit in greatly clarifying and refining, what educational 
practice has already known and applied in a less disciplined way. Applied 
research, knowledge-based development, and evaluation are richly inter- 
dependent and do contribute to each other. The problem for basic educa- 
tional research may as well be that it contributes little and receives 
little in return from these other connected, practice-oriented activities. 



Development in Relation to Research and Evaluation 

Turning to Glass and Worthen's treatment of the relationship of 
development to educational research and evaluation, I find full agree- 
ment with their view of the dependence of development upon sophisticated 
evaluation. Traditionally, the largest effort at educational develop- 
ment has been by commercial publishers and by schools themselves. 

Havelock (1969, pp. 3-25) has noted that these organizations both his- 
torically and structurally are Isolated from the mainstream of scientific 
knowledge and they rarely attain the Internal competence to seek out 
and effectively use scientific sources of information. Development that 
is knowledge-based as well as systematic, rigorous, and empirical is a 
much more recent activity in public education.^ 

I find complete agreement with the authors' point that the laws of 
the social and behavioral sciences are of extremely limited generality 
(p. 93), but I wonder why they fail to conclude, based on their own 
discussion of "universality" (pp. 84,86), that there is relatively little 
in educational research that can approach the pure type they have char- 
acterized, i.e., "relatively permanent, applicable to schooling nearly 
everywhere, . . . should subsume a large number of instances of teaching 
and learning" [p. 86 ]. 

Their discussion of how basic and practical knowledge can be related 
is provocative but may in part miss the point by underemphasizing the 
role played by applied research and technology. Undoubtedly basic know- 
ledge does have a heuristic relationship to educational development. 

They have emphasized that: 

" Basic knowledge will never prescribe a 
particular development, but it will stimulate 
creative minds who understand the basic know- 
ledge to develop materials and practices based 
loosely on it. If the innovation is developed 
and evaluated with the best empirical methods 
of evaluation and applied research, the payoff 
from basic research can be realized more 
quickly at a fuller value" [p. 99 ]. 



^Rigorous development In military and industrial training programs 
is at least 30 years old. Glass and Worthen, like so many other writers, 
seem unaware of the extent to this effort. If educational development 
is at present in a state of conceptual deprivation (Schutz, 1968, p. 3), 
then it may as well be due to the parochialism of educational researchers 
and administrators who have recently become educational developers and 
managers of development (Gagne, 1962; Glaser, 1962; Stolurow, 1964; 
Eckstrand, 1967; Uhlaner, 1967; Smitb> A968. ) . 
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But I should have liked to see them also underline the statement: 
"It seems vain to hope that basic knowledge will be implemented with- 
out applied research and evaluation to test under special circumstances 
that which basic research only suggests may be true" [p. 100]. Despite 
this cavil about emphasis, the entire section discussing the relation- 
ship of development to educational research and evaluation [pp. 96-101] 
is, in my opinion, a distinct contribution to the literature. 



Diffusion and Inquiry 

Possibly the least satisfying part of the entire paper is the dis- 
cussion of diffusion and its relation to other inquiry related activities. 
The treatment is both brief (four pages) and superficial. 

In this critiquer’s view, educational diffusion may become the 
Achilles* heel of the entire research, development, and evaluation 
enterprise. I agree with the authors* view that despite the impressive 
body of literature accumulating with regard to diffusion, there are 
relatively few studies that tell us how diffusion actually occurs or 
what specific skills or knowledges are essential for adequate perform- 
ance of diffusion roles. But this is troubling for more reasons than 
the possible misallocation of funds to train diffusion personnel. If 
the reason for investment in DD&E is practice improvement, then we must 
make a major effort to study, understand, train, and develop a diffusion 
capability. Educational diffusion may not be a disciplined inquiry per 
se, but it sorely needs to be the subject of considerable disciplined 
inquiry. Conversely, disciplined inquiry critically needs improved dif- 
fusion to strengthen its own operations and to provide practical justi- 
fication for the investments in research, development, and evaluation.^ 



Federal Policy 

We have noted that there are in fact two essays within this one 
paper. The second, titled "Toward More Useful Implementation of Educational 
Research, Evaluation, Development, and Diffusion*' first discusses federal 



Simon Rottenberg (1968) , in his article "The Warrants for Basic 
Research", notes that to know whether basic research should be under- 
taken or supported as a productive investment , one can only examine 
past experience and extrapolate to future success. He further observes 
that calculations of net success and failure values done for basic 
research in different disciplines and subdisciplines would undoubtedly 
give different distributions. Given Glass and Worthen's italicized 
comments on payoff on basic research, and their view of the lack of an 
adequate knowledge base in diffusion, one wonders how past experience, 
extrapolated to the future, could lead to other than rather dismal esti- 
mates of the value of educational research unless it is "bridged" by much 
more effec.tive applied research, dc^.velopment , diffusion, and evaluation. 
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policy and then makes some proposals for organizing RDD&E so that they 
might be more potent In shaping schooling. Regarding federal policy, 
they express concerns that disbursement of federal funds to universities 
for R&D runs counter to the organization of applied R&D In the 
country and that It threatens to subvert the unique contributions which 
the universities can make through basic research. Citing National 
Science Foundation data, they ask, "If applied research and development 
In other fields are conducted largely within private Industry why should 
applied research and development In education be conducted In universities?" 
[p. 104]. 

Well, first of all, applied educational R&D Is not entirely conducted 
by universities. Possibly much more than one-third of the rigorous work 
on educational (and related training) R&D Is performed by educational 
laboratories, other nonprofit agencies (such as EDC, AIR, SRI, or HumRRO) , 
and Industry. Second, close examination of their Table 1 reveals, that 
In 1953-54, 44% of the sources of funds for all applied R&D came from 
Industry. In 1957-58 the figure Is 34%. The point Is that private 
Industry makes very substantial Investments In supporting applied R&D 
(primarily In the physical sciences) and has thus developed a signifi- 
cant capability that can be exploited by federal funding. Would the 
reader hazard a guess as to how much Industry has Invested of Its own 
money In educational R&D? Third, note that the total Investment for 
1968-69 Is $13,701 millions for applied R&D and another $2,146 millions 
for basic research for a grand total of $15,847 millions. One estimate 
by the U. S. Office of Education (Gldeonse, 1969) would place all ed- 
ucational R&D for the same period at approximately $250 millions or 
less than l/63d of the grand total for all R&D. 

The point Is that applied educational R&D Is a very small part 
of the total R&D enterprise. If one examines the other small fractions 
of this total that are associated with the behavioral sciences, a.g., 
psychology, sociology, anthropology, he will again find that, as for 
education, most all of the funding Is federal, and that universities 
and nonprofit agencies are the primary recipients. Finally, as Glass 
and Worthen have noted, there are marked differences between scientific 
knowledge In physics and the social and behavioral sciences. "Social 
scientific laws are little more than probability statements about the 
tendency of one variable to be related to another. These laws are highly 
Interactive with factors auch as geography, time, cultures, and individ- 
ual characteristics" [p. 97]. These differences may explain why 
not only the Office of Education, but also the Department of Defense, 
and most other government agencies have not turned to industry in funding 
applied R&D efforts directed toward educational, training, or other 
social problems. 



Organizing RDD&E 

Regarding the authors' views that greater payoff can be realized 
from educational research If (a^) It adopts a more interdisciplinary 
perspective, (^) Its relationships to practical problems of development 
Is examined critically, and (c) It Is given financial support, I cannot 
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argue with point (a); but» I have reservations about both points (b) 
and (c). The problem with respect to these last two points Is that 
the view taken Ignores serious allocation and organizational problems. 
Clearly » over the past decade there has been a marked shift In funding 
from research to development. In the coming decade this trend will 
undoubtedly continue, along with a substantial (relative) Increase In 
allocations for diffusion (Clark and Hopkins, 1969). Moreover, there 
Is little doubt that educational R&D has not recently enjoyed much 
support for Increased funding from either the Congress or the Administra- 
tion. Financial support may only be forthcoming when educational RDD&E 
becomes articulated, organized, and achieves an allocation of funding 
among Its activities that leads to a demonstrable, significant Increase 
In the Improvement of educational practice. 

I heartily agree with the authors' observation that the reward 
structure within universities Is a major Impediment to the effective 
utilization of educational evaluation talent, and with their observa- 
tion that there Is probably a greater chance of creating new Institutions 
as alternatives to the universities than of altering the reward structures 
of universities. The educational laboratories are examples In the develop- 
ment area, and the "evaluation teams" of local educational agencies are 
examples In the evaluation area. 



Accountability and/or the Market Mechanism 

Glisj and Worthen propose a patently simple and "utopian" solution 
for the problems of development and diffusion, i.e., to move schooling 
and most of development and diffusion out of the grants economy into 
the private sector of the economy and under the Influence of the market 
mechanism. They argue that the motivation for developing and disseminat- 
ing educational products must originate with schoolmen and that this 
motivation Is lacking as long as schoolmen are spared the obligation 
to maintain a market relation with the public. They note that funds 
are granted to the educator (grants economy) and he Is not required to 
present feedback to the grantor on how well the funds are being spent. 
Therefore, the first step In Instituting a market mechanism Is to make 
the schoolman accountable for his successes and failures* (This crltlquer 
notes this should also generate a strong demand for evaluators.) The 
next step would be to create competition, real options for parents and 
children. They note that it would be foolhardy to underestimate the 
practical problems of shifting schooling even a small distance on the 
grants — exchange continuum, but advance the notion in principle that 
development and dissemination would proceed more rapidly and to greater 
effect if carried out In the private sector. 

It might. It seems more realistic, however, to anticipate move- 
ment toward increased accountability for educational practice, es well 
as for educational research, development, diffusion, and evaluation, 
all primarily within the grants end of the economy. Solutiox's as to 
alJ'ncations and relationships among education RDD&E and practice will 
be dictated by pluralistic political and social pressures and influences 
far more complex than even those of an open market. 



Educational practice and the research, development, evaluation, 
and diffusion that would seek to Improve It are "social goods," rather 
than "private goods" In the economic sense. Glass and Worthen are right 
In their concern for "accountability." But all of these activities 
need to be accountable to something more than a market generated simply 
by the demands of parents or their children. 
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RESPONSE TO THE CRITIQUE 



Gene V Glass and Blaine R. Worthen 

Hood has written a most comprehensive and analytic critique of our 
essay on educational inquiry. His critique exhibits good understanding 
of the major points of the paper and of the more subtle points we atttmpted 
to make; such a critique needs little rejoinder and we will restrict our 
comments to attempt to clarify points which Hood’s critique has shown 
were not clearly communicated in the original paper. The reader is correct 
who assumes that we are in agreement with any of Hood’s criticisms we do 
not address below. 

Hood is disconcerted by the fact that development and diffusion 
receive relatively less attention in our essay than do research and 
evaluation. He is especially concerned that diffusion is viewed as 
lying . . totally outside the area of disciplined inquiry. . (p.l24) 

and notes that it is unfortunate that our ". . .diagrams regarding dis- 
ciplined inquiry were not extended to include explicitly the domain of 
educational practice” (p.l24). Hood has proposed a revised diagram 
(Figure 1, p.l25) in which he has juxtaposed spheres of concern for ed- 
ucational practice and disciplined inquiry, resulting in a portrayal of 
diffusion as directly relevant to educational practice. We have no 
argument with Hood’s i. agram or his stress on educational diffusion as 
an important activity. However, our treatment of diffusion is ’’brief 
and superficial” because discussing inquiry-related activities was not 
one of our purposes in writing the paper. The charge to us was to con- 
ceptualize the domain of educational inquiry and discuss interrelation- 
ships among components of the inquiry process. As a result, diffusion — 
a noninquiry function — appropriately received little direct attention. 

Hood seems to conclude from our statement that diffusion is not inquiry 
per se , that we feel it is unimportant or of less worth than research 
and evaluation. Such was never our intent. Diffusion is clearly critical 
if the results of inquiry are to have any impact on practice. However, 
diffusion is not thereby an inquiry activity any more than reading is an 
inquiry activity because research reports must be read to be used. Hood 
stated that ’’Educational diffusion may not be disciplined inquiry per se , 
but it sorely needs to be the subject of considerable disciplined inquiry” 
(p.l30). We agree, but to claim that diffusion must be the object of 
inquiry is not to say that it is inquiry itself. 

Hood has expressed concern with our unwillingness to extend the 
term ’’evaluation” to include ’’decision-centered” evaluation activities. 

Our concern is not whether evaluation is decision-centered (hopefully 
all evaluations will lead to sound decisions) but whether various ’’de- 
cision-centered” activities are genuinely evaluative. Context description 
and process monitoring are oftv'.n essential to an evaluation, but in and 
of themselves they fall short of being evaluations. To reserve the term 
’’evaluation” for activities where explicit valuation occurs does not 
negate the Importance of evaluation-attendant activities which are not 
directly evaluative. 

Hood states that we . .propose a patently simple and ’utopian’ 
solution for the problems of development and diffusion, i.e., to move 
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schooling and most of development and diffusion out of the grants economy 
into the private sector of the economy and under the Influence of the 
market mechanism" (p.l32). We did not Intend to propose a solution for 
actual Implementation at all. We meant only to propose an "explanation" 
of the tortolse-llko. Improvement of the schooling system. We are not 
optimistic that "free-market" schooling ever could be Instituted In this 
country, but we feel that the absence of any of Its features may partly 
explain the lack of attention to development and evaluation In our 
schools. We expect many more years of the present economic structure of 
schooling and have no practical alternative to propose; we merely Intended 
our analysis of an alternative economic structure to serve to Illuminate 
problems that exist In the present structure. If we were to write the 
section again, we would be tempted to sprinkle the term "utopian" liberally 
throughout our treatise, as Hood would have It. 

We applaud Hood's thoughtful critique. 
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SYNOPSIS 



The paper by Briggs essentially picks up where the Glass-Worthen 
paper leaves off. Its central focus is educational development and 
diffusion, and in it Briggs describes a consortium based model of 
educational development that Is designed to assure not only the pro- 
duction of good educational products, but their diffusion as well. 

In addition to substantive focus, the paper also differs from the 
Glass-Worthen paper in style and point of view. In contrast to the 
rather formal style of exposition adopted by Glass and Worthen, Briggs 
uses a more personal, informal, experiential style. Briggs also con- 
ceives of his model of development as "research based" — a conception 
that directly challenges the Glass-Worthen position of development as 
being essentially an extension beyond the domain of disciplined inquiry. 

Briggs identifies 17 steps in the model that he describes. These 
range from the setting of educational goals and objectives to the prepara 
tlon of pre- and posttest measures, the development of instructional 
materials, and the preparation of teachers to use the assessment and 
instructional materials. The notion of cooperative decision making in 
the operation of the model is consistent with Gldeonse's notion of par- 
ticipatory decision making in the operation of a market model (see 
Chapter 1) . 
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DEVELOPMENT AND DIFFUSION AS MECHANISMS FOR 
EDUCATIONAL IMPROVEMENT! 

Leslie J. Briggs 

Florida State University 

Introduction and Overview 



In the examination of various means by which educational changes 
and improvements have been or in the future may be brought about, the 
author concurs with Schutz (1970) that: 

**While there is a vague realization that it should be 
possible to translate available relevant knowledge into 
a form that permits improved educational practice, it is 
erroneous to assume that the means for doing this are 
presently clear” [p.39]. 

The place of development and diffusion as systematic activities for 
bringing about educational Improvements is in a youthful stage, exper- 
iencing rapid growth, but not yet being fully realized. An underlying 
rationale for this paper is the author's belief that the nature and 
scope of educational development, as a set of science-based activities, is 
still to be defined; and, the concept of educational diffusion is still 
searching for meaning and application. 

For purposes of entry into the topic of this paper, the definitions 
of development and diffusion offered by the National Center for Educa- 
tional Research and Development (Gideonse, 1969) are recognized. 

Educational development is defined as: 

. . (the production of) materials, techniques, processes, 
hardware, and organizational formats for instruction. The 
basis of such development is our knowledge about learning, 
motivation, instruction, and education. The materials and 



^ The absence of the words "research” and "evaluation" in the title of 
this paper is intentional. Since other papers in this series heavily 
emphasize the functions of research and evaluation in bringing about 
educational improvements, it was intended that this paper focus more 
directly upon the contributions of development and diffusion. There- 
fore, other papers in this series will be relied upon to present many 
"ground-clearing" definitions and theoretical positions of research 
and evaluation to development and diffusion. Formal definitions will 
be minimized in favor of descriptions of what has been done and what 
might be done in the future. , 
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techniques developed are designed to accomplish certain 
objectives, specified in advance, which are construed to be 
part of the broader goals of Instruction or education. • . 

Examples of development in education include the construction 
of programmed instructional materials, the building of curri- 
culum units, the designing of computer-assisted and computer- 
managed instruction, and the validation of teacher centered 
instructional techniques based on our knowledge of teacher- 
pupil interactions. Careful development of television pro- 
gramming, the construction and validation (as to learning effect) 
of single concept films, and the development of new organiza- 
tional forms for schools and universities are further exam- 
ples" [pp. 3-4]. 

Similarly, Borg (1970) has pointed out that the two goals of development 
are: (ai) to produce products whose effectiveness has been sufficiently 

established, and (^) products in a form which are ready for operational 
ins tallation in the classroom. Comparable definitions of development 
have been proposed by Hemphill (1969) and Schutz (1970). 

Diffusion has been defined by Gideonse (1969) as: 

". . .the entire process by which innovations are spread 
throughout a culture, a profession, or some other expanded 
social system. Dissemination mechanisms may be a key factor 
in the diffusion process, but so might the active support of 
rigorous development or the provision of adequate support for 
research" [p. 4]. 

Following the pattern set by this definition, educational diffusion* in 
this paper will refer to all means of bringing educational developments 
into effective classroom use on a widespread scale. Thus, dissemination 
becomes one means of diffusion. Other means are formal and inservice 
teacher training, demonstration, installation, utilization of consultants , 
workshops, seminars, and all messages conveyed by any media designed to 
promote the adoption or utilization of educational information or products. 
Definitions proposed by Jung and Lippett (1966), Havelock and Benne (1967), 
Farr (1969), and McClelland (1969) offer support to these dimensions of 
diffusion activities in education. 

Two general questions are considered within this paper: (^) how do 

development and diffusion activities typically take place in education, 
and (^) how might their purposes be better achieved? 

The primary aim of the paper is to present a view of development 
and diffusion activities that will assist in making them more effective 
for the improvement of education. A supporting aim of the paper is to 
describe selected considerations that have contributed toward this pro- 
posed view of educational development and diffusion. 

This paper is not intended as a definitional or theory-generating 
treatise, or as a comprehensive or exhaustive review of the "state-of- 
the-art" in educational development and diffusion. Rather, the paper 
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is intended as a selective and illustrative view of the purposes and 
means of educational development and diffusion as they have taken place 
in the past, as they are currently practiced, and as they might occur 
in the future. This view is based primarily upon the author's experi- 
ence with development and diffusion efforts over the years in many con- 
texts. 

The remainder of this paper is organized in five sections. The 
first section is a summary discussion of several alternative means or 
"mechanisms^' which the author sees as available for bringing about educa 
tional change, methods which they may employ to bring about change, and 
resources which they may use to support change. The author's preferred 
mechanism for educational change is highlighted and discussed in terms 
of the process of change. 

Following this is a summary and analysis of various ways in which 
educational change and innovation have been looked at in the past. The 
literature review of Havelock, Guskin^ et al., (1969) is sum- 
marized around "models" of diffusion (dissemination and utili- 
zation) which Havelock, et al. identified during their survey. These 
"models" of diffusion Include the research-developroent-dif fusion model, 
the social interaction model, the problem-solver model, and the linkage 
model. A critical reaction to these models is presented, focusing on 
their deficiencies for describing educational change and innovation. 

Section three in the paper attempts to identify guidelines emerging 
from successful development and implementation efforts of the past. 
Prototypes of past successful development programs are discussed, in- 
cluding examples from the military, industrial, and educational settings 

The fourth section is an examination of educational development as 
it is currently practiced, including those who are involved with educa- 
tional development, the kinds of developmental activities which are 
performed, and techniques which are employed. Three major areas of 
educational development are examined: commercial publishers, curriculum 

development projects, and professional researchers-developers . 

The final section of this paper, to which each of the four pre- 
ceding sections contribute and support, is the proposed model for 
development and diffusion in education. Section five includes an 
identification of the basic assumptions underlying the proposed model, 
a description of the general components which comprise the model, a 
discussion of factors to consider in applying the model, an analysis of 
elements which will effect the success of the model, and some notes on 
organizational change advocated by the model. 

The paper begins with the author's view of educational change and 
innovation, including mechanisms for and the process of it. 
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Educational Change and Innovation 



There Is a rapidly growing literature which deals with major edu- 
cational Improvements that have been brought about by the dissemination 
and utilization of knowledge and products. Havelock, et al. (1969) 
have Indicated that this literature on educational Innovations, con- 
sisting of about 100 studies In 1955, grew to over 4,000 studies In 
i960. Undoubtedly, It Is much larger now, Havelock estimates 10,000 
references at the start of 1969. 

After a thoughtful review of dozens of reports In this literature, 
this writer has concluded that there Is no singular means by which 
changes have been produced In the past and are being produced at the 
present time. Therefore, the writer advocates a different view of 
educational change and Innovation from that appearing In the literature. 
Seen In a broad perspective, and underlying this view. Is the hypothesis 
that specific changes In education depend largely upon three factors: 

(^) who wants the change, (^) what general method is used for producing 
the change, and (O what resources for change are employed. To show how 
these change-related factors may interact, a few examples are considered 
in terms of how not one but several mechanisms may be used for bringing 
about educational change . 



Mechanisms for Change 

Mechanism, as used in this paper. Is defined as any means for com- 
bining activities and resources to accomplish a definable goal or 
objective. Alternative mechanisms for educational change are described 
In the following paragraphs, starting with the consideration of who 
wants change . 

1. When an aroused socle ty wants change In education. It seeks 

a shift in the finances which support federal programs, a reform In the 
management of federal or state educational agencies, or the Initiation 
of new policies through recall, referendum, and legislation. Some 
methods may be quicker than others, but the principal result Is the 
same: new persons assume power and responsibility. These new power 

figures then know that their retention of authority rests upon making 
the kind of changes promised or desired. 

2. When a concerned community desires educational change, it 
modifies either the personnel or behavior of the school board and the 
school administration. When the desires of community are the same as 
those of the local educational leadership, rather radical changes have 
taken place peacefully In a relatively short time, as described by 
Esbensen (1968) . 

3. When a local teacher desires change, he confers with other 
teachers and local sources (knowledge ’’linkers”) who keep abreast with 
changes in outside institutions, such as universities or research cen- 
ters. He also seeks information from professors, researchers, industry. 
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or other sources, looking to find the technique or product that appears 
to suit the desired change • In some Instances the teacher may develop 
his own Innovation, but the Innovation Is not likely to come Into wide- 
spread use because of lack of an organization to diffuse the Innovation. 

4. When a researcher wishes to Institute change, he has typically 
attempted to bring It about by dissemination of Information, by persuasion, 
and by personal contact with t lachers and schools. As Borg (1970) has In- 
dicated, neither the local teacher-made Innovation nor the dissemination 
attempts of the Individual researcher have been spectacularly successful. 

5. When an Industry wants to bring about change In education. It 
most commonly takes a marketing approach to use a product as the actual 
change agent. Since products are easier to diffuse than are Ideas or 
research findings, the marketing approach has been more succe 5 :sful than 
some of the others just mentioned, but It Involves the risk of self- 
serving rather than public-serving. It also finds Its appeal more 
directly to the judgment of the teacher-customer than to the public the 
teacher Is supposed to be serving. For this reason. In the future a 
new mechanism may be created by which the public sets the general goals 
of education, the educationists (Including the researcher) translate 
these to needed product specifications, and Industry produces the de- 
sired products (Briggs, 1970b). Variations of this aspect of marketing 
can serve the needs of the public and of Industry. 

6 . When a combination of educational and research Institutions 
want to bring about change, the consortium approach appears to be a 
most promising and satisfactory mode of change. Such consortia can In- 
clude schools, universities, policy research centers, educational lab- 
oratories and research centers, and/or Industry. A consortium can be 
sensitive to the wishes of the public as to Its desired goals of educa- 
tion; It can utilize the expertise of researchers In deciding upon the 
kinds of products and processes needed to reach the goals; and. It can 
utilize the resources of Industry to produce and Install the desired 
products. It also has the potential for coordinating the new processes 
and products with both the preservice and the Inservlce training of 
teachers. Furthermore, It has the multlorganlzatlonal resources which 
make possible a comprehensive orientation of evaluation, field testing, 
and product Improvement through formative development. Such efforts 
can Involve federal agencies, such as the U.S. Office of Edu^-atlon, or 
the proposed National Institute for Education (NIE) , and can become 
the means for Increasing the effectiveness of State departments of edu- 
cation. The consortium approach to educational change can bring about 
organizational self-renewal to operating agencies; better goal direction 
to research agencies; more effective use of Industrial potential; and 
most Important, educational accountability to the public served. 

The foregoing discussion of various mechanisms which may be used 
for bringing about educational change are summarized In Table 1. 
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TABLE 1 



A Sunraary of Some Mechanisms 
and Related Factors for 
Bringing About Educational Change 



Who Initiates 


Method(s) 


Resources 


Mechanisms 


!• Aroused 
society 


Revision of federal 
programs. 

Reformation of ed- 
catlon agencies. 
Initiation of new 
policies 


Accountability , 
Stimulus for 
self-renewal 


New Personnel, 
Fiat 


2. Concerned 
community 


Personnel changes or 
revitalization 


Redefinition of 
priorities , 

New Methods 


Improved 

s choo 1- commun 1 ty 
communication 


3. Local Teacher 


Seek out knowledge 


Local knowledge 
"linkers” , 
Professional 
societies , 
Universities , 
Information 
centers 


Invention , 
Adoption , 
Application 


4. Researchers 


Persuasion, 

Publishing, 

Teaching, 

Consulting 


Research infor- 
mation , 
Publications , 
ERIC system 


Dissemination of 
Information, 
Translation of 
research findings 


5. Industry or 
Curriculum 
development 
teams 


Production , 
Marketing , 
Development 


Capital, 
Organization , 
Equipment , 
Personnel, 
Writers 


Salesmen, 
Advertising , 
Promotion , 
Teacher training 


6. Combination 
of Education, 
Industry, 

RSiD agencies 


Consortia for: 
Research, 
Development , 
Diffusion, 
Installation , 
Testing 


Public contact, 
Expertise, 

Models , 
Knowledge, 
Experience, 
Multiple funding 


Systems analysis. 
Development models, 
Teacher training. 
Research informa- 
tion , 

Team development. 



Cooperation of 
school and industry 
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Process of Giange 



Within the body of literature on educational change and innova- 
tion, one finds the recurring assumption that educational change is 
essentially a persuasive process by which research knowledge gets 
disseminated to the teacher, who first is aware, then interested, and 
then persuaded to try and then adopt a classroom procedure which is an 
application of a research finding. But in the opinion of this writer, 
precious few improvements are made by this process, and the effort de- 
voted to cajoling, pleading, coaxing, and explaining is too out of con- 
text for most teachers to make the application. 

Therefore, some insist that what is needed is translation of the 
research finding into language which the teacher can understand. But 
the viewpoint taken here is that we do not so much need translators as 
we do educational engineers who can apply research findings in the de- 
sign of workable (and testable) products, techniques, tools, or instruc- 
tional packages. 

Just as this writer sees little to be accomplished by translation, 
in and of itself, he also sees little need for an isolated practitioner 
of diffusion. The process of creating widespread and significant change 
is seen as more nearly requiring an approach similar to the consortium 
method. Through consortia, plans for installing, testing, and institu- 
tionalizing change are built into the development-utilization plan by 
prior arrangements of member organizations of the consortium. 

The writer does not wish to be thought of as having summarily dis- 
missed the bulk of literature in disseminating, diffusing, installing, 
testing, and institutionalizing educational change. However, he has 
seriously and thoughtfully concluded that the best context within which 
to refer to these functions is in a consortium-based model, such as the 
one presented later in this paper. 

Within such a model, the researcher's role is not restricted to 
doing research. He designs, consults, interacts, and assists in many 
aspects of the development-utilization continuum. Many researchers, so 
trained, are now excellent developers. In fact, the most skilled deve- 
lopers may be reseachers who are also able to manage a complete research 
to installation/utilization effort for an entire consortium. 

Another reason the consortium arrangement for educational change 
is highlighted in connection with development and diffusion in this 
paper is that many alternate versions of how research aids development 
and diffusion are self-defeating. That is, sometimes researchers, 
developers, and other team members of a curriculum development effort 
set out to design a set of materials that will have the widest possible 
acceptance by teachers who are uninfluenced by an aroused society, a 
concerned locality, or a dynamic school administration. Under those 
arrangements, only gradual, step-at-a-tlme changes can be sold to the 
teachers. Such projects may cost as much for a small, widely-accepted 
change as a project bringing great change would cost under a prior-agree- 
ment, consortium-based arrangement. This is not greatly different from 
approaches used by industry, whereby only such changes are introduced 
as can be sold to a wide range of unorganized, separate school systems. 
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While a product easier to sell than an idea, a product so sold to 
unorganized masses of teachers must not be greatly different from 
products they are now using. 

This is not to say that it is unworthy to introduce small changes 
to groups of teachers by a curriculum development team or by an Indus- 
trial approach. It is to say that it is precisely when there is not 
an aroused public, and not a concerned community, that the industrial 
approach may be most appropriate for small change. 

Since minor changes made acceptable to unorganized masses of 
teachers is so similar a process, whether done by a curriculum develop- 
ment team or by industry, the industrial marketing method will not be 
further discussed here, nor differentiated from the curriculura/develop- 
ment team method. The persuasive or marketing aspects are similar, 
though the curriculum/development team may carry more prestige and get 
bettv^r cooperation in training teachers to use the product. 

It xc to be hoped that this discussion of the curriculum/develop- 
ment team approach is not construed as an indication that such teams 
of professional people cannot produce products superior to those produced 
by industry when operating in its conventional, solitary role. It is, 
therefore, necessary to distinguish between the intent of the ground 
rules. 

If a professional development team sets out to produce the best 
product it can and to influence (and train) as many teachers as possible 
to use the product, but in no case to compromise the product based on 
estimates of large scale acceptability, then quality and amount of im- 
provement have priority over sales. If sales are placed first, then 
quality or change may be in danger of being compromised, with little 
difference in the product whether developed by Industry or by a pro- 
fessional team. However, to develop an optimum product, which will be 
adopted, advance support is a most important ingredient. Again, the 
consortium-based approach is seen as one effective way to get this ad- 
vance support . 
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Planning for Educational Change and Innovation 



2 



Havelock and others (1969) have achieved a most recent and compre- 
hensive review of the diffusion literature in their report entitled 
Planning for Innovation; A Comparative Study of the Literature on the 
Dissemination and Utilization of Scientific Knowledge , 

The very title of the report by Havelock and his co-workers, as 
compared to the title of this paper, reflects the two different stances 
taken in the two documents. Havelock was reviewing at an earlier date 
and he was intentionally writing a review, not a ’’position paper.” He 
was thus reacting to existing models of change, not presenting a model 
of his own. With these considerations in mind, the fact remains that 
his view of the research-development-dif fusion model for change seems 
to differ markedly from the stance taken by the present writer. His 
evaluations were probably a reflection of the amount of empirical data 
available in the literature he reviewed for the models he was comparing 
and contrasting. However, his reviews of change as a function of dissem- 
ination and utilization of knowledge differ from the present writer’s 
view of the development and diffusion of products (placed in the con- 
text of accountability and organized effort) as the most powerful view 
for creating change and innovation in education. Furthermore, the pre- 
sent writer feels that Havelock’s report neglects some of the important 
features of the research and development process and how they support 
the diffusion enterprise. Perhaps these features do not appear as clearly 
in the literature as they loom promin^ at :’.n the thinking of a practitioner 
of research and development. 

Nonetheless, Havelock’s review has performed a great service in 
providing in a single volume the substance of a literature which is so 
widely scattered in a large number of sources. The present writer 
responds to that report in two ways here. First, a very brief noncriti- 
cal summary of Havelock’s findings is presented as he has organized and 



^ The list of References reproduced in this paper does not indicate the 
extent of the sources consulted. Essentially, three independent samples 
of the literature were used. Havelock’s review is, of course, the most 
comprehensive source that was read- Second, a large collection of papers, 
assembled by the project staff at Teaching Research in Oregon, was made 
available. Third, a locally-initiated ERIC search was made. The writer 
felt that these three ways of viewing the literature all gave about the 
same overall account of approaches to educational diffusion. Little new 
was encountered during the later phase of the reading done by the writer. 
Since the writer has taken this particular stance in relation to the 
literature, he saw less need for more detailed citations. The purpose of 
this section of the paper is to react to the literature rather than to 
document it in detail, so the references do not at all constitute a 
history of research on diffusion. Havelock’s review is closer to that 
function, and so is his reference list a much better acknowledgment of 
the work of the people who have labored the most in this area of research. 
Since his review is available, perhaps no further apology is needed con- 
cerning the absence of names of those workers in this paper. 



evaluated them* Second, the vrriter offers comments on Havelock's find- 
ings from the standpoint of a supporter of research and development in 
education. 



Havelock *s Review 



The summary, synthesis, conclusions, and implications of the Havelock 
study are presented in Chapter 11 of the project’s final report. He first 
comments upon the following features of the literature surveyed: large 

quantity, rapid growth, wide scope, large proportion of empirical studies, 
and wide dispersion of sources. 

In quantity, he found the volume of literature to exceed his initial 
expectations. The field of educa*. Ion accounted for 17% of the sources: 
agriculture accounted for 13%, and communication accounted for 13%. 

Rate of growth of the literature was viewed as impressive as the 
quantity . 

The scope and variety of the literature led Havelock to conclude 
that the two words dissemination and utilization "covered a lot of 
territory" [p. 11-2]. One source of great variation in the dissemina- 
tion/utilization studies was size of adopting unit; another was wide 
perspective from macroscopic to microscopic; and, a third was range in 
type of study — experimental, case study, field research, and theory. 

The scope of contexts covered curriculum diffusion, administrative 
arrangements, organizational change, technology diffusion, and new roles 
and organizations* Variety was also found in studies classified as re- 
search, development, and "practical wisdom." 

Within this chapter, Havelock also presents an overview and syn- 
thesis of four models for the dissemination and utilization (diffusion) 
of knowledge. In the four subsections to follow, his overview is sum- 
marized. Please note in these summaries that the four models reviewed 
in the literature by Havelock differ greatly in the stress upon develop- 
ment . While the view favored by this writer places development and 
diffusion on a par level in the research-development-diffusion model, 
this is not always characteristic of the three other dissemination/utili- 
zation (diffusion) models. Also note, as previously discussed, that the 
three other models view change as taking place by the utilization of 
knowledge , while the research-development -diffusion model views utiliza- 
tion of developed products as the principal change-agent. It is obvious 
that knowledge is utilized in the research-development-dif fusion model, 
but also products and evaluation loom large in the picture. 

The research, development, diffusion (RD&D) model . This model is 
broken down into different numbers of subcomponents or characteristics 
by different writers, but Havelock (p.11-5) sees these all reducing to 
five features: 

1. Dissemination and utilization should be a rational process, 
requiring a rational sequence of activities from research 
to diffusion. 



2. Planning in a massive scale must be employed to coordinate 
the entire process. 

3. Division of labor is necessary due to the massiveness of the 
process, resulting in a necessary separation of roles and 
function. 

4. A passive consumer is assumed to be the target audience; he 
will accept the innovation if it is delivered skillfully. 
Scientific evaluation at every stage of research, development, 
and diffusion is supposed to bring about acceptance. 

High initial development costs are assumed, and are justified 
because of the larger long-run gains in terms of efficiency, 
quality, and massive utilization. 

Proto tpyes. of the RD&D model are seen in industry, defense, space, 
and agriculture. In education the following proponents were cited: 

Henry M. Brickell, Egon Cuba, and David Clark. This model is cited as 
useful at the macrosystem level because it provides for multiple organ- 
izations and roles, thus linking research, product, practitioners, and 
consumers. It is also seen as a major basis for policy in the U.S. 

Office of Education. Acknowledgment is made by Havelock that while flow- 
charts depicting this model often imply a linear flow of actions and in- 
formation, feedback and redesign are in fact characteristics of how the 
model actually works. Yet, the linearity of the process as depicted by 
Havelock overshadows these potential interdependencies. 

Although Havelock notes the RD&D model to be ”a very useful and 
relevant paradigm for technical and social change" [p. 11-5], his basic 
criticism is that the RD&D model is ". , . over-rational over-ideal- 
ized, excessively research oriented, and Inadequately user oriented. . 

[p. 11-7]. Havelock cites Frank Chase for suggesting that Cuba and 
his colleagues have spelled out the model so clearly that it gives other 
educators something to criticize or to avoid by design of alternate 
conceptualizations . 

The social interaction model . This model, growing out of work by 
Barnett in anthropology, does not deal with research and development, 
per se, but it assumes them as a precondition for study of the diffusion 
process. This model has little to say of how science and technology 
may function to bring about new products. Rather, it deals with products 
which are in existence. A typical example involves innovations which 
are in a concrete, diff usable form, such as an audiovisual device. The 
innovation is hence a stable product whose flow through the social sys- 
tem can be studied. This flow is studied in the form of social struc- 
tures and relationships of the people involved. 

According to Havelock, the theory and research supporting the model 
stress the following: (^) the importance of the social relations net- 

work, (^) the importance of personal relationships, (c) the user’s 
position in the network, (^) the importance of group identifications, 

(£) the irrelevance of size of adopting unit and (O different influ- 
ences' import at different stages in the process# 
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This model acknowledges the relatively minor role that publications 
play, as compared to face-to-face contacts among people, in obtaining 
dissemination to a passive audience. Given the passive audience, the 
importance of human interactions of a complex nature is emphasized. In 
this model "opinion leadership" exerted by members of the target audience 
is a major influence upon adoption. The prestige of such persons, their 
norm-setting behavior, and their frequent interaction with others inside 
and outside their own group are important. It is also in this model that 
appear the "innovator," the "laggard," and the "early majority." 

With relatively stable and easy-to-use innovations like audio- 
visual devices, one can see hew the social interaction model can embrace 
the work of the researcher, the manufacturer, the field representative, 
Che early user, etc. In applying this model, it would appear that many, 
such as classroom teachers, are more influenced by what their peers are 
using than by results of research and testing conducted by the "expert." 
In some instances word-of-mouth communication appears to do better than 
research reports, for example, in the field of agriculture where the role 
of the counuy agent is important in helping obtain the early adoption 
of developed products . 

The work of Theodore Newcomb is cited as significant in revealing 
how firm roots in a culture in early life can result in persistence of 
old attitudes in spite of exposure to new attitudes and informa"“ion. 

Firm belongingness to reference groups leads members to retain the 
group's value system. On the other hand, innovators move from group to 
group, enabling them to see personal value in things not valued in some 
of the groups. Havelock points out that if large numbers of people 
maintain diverse contacts with differing groups, a highly innovative 
society would result. One merit of research relating to this model is 
that while person-to-person contact is important, results have been 
studied for all sizes of groups and organizations of diverse kinds. 

The stages of adoption on the part of a person or organization, 
according to this model, are: awareness, interest, evaluation, trial, 

and adoption. Different influence strategies are effective at the vari- 
ous stages. The effective media studied for the different stages of 
adoption include mass media, demonstration, contact with experts, and 
informal contact with peers, respectively. 

While, historically, research on this model began in anthropology, 
later key influences cited by Havelock in social psychology were made 
by Newcomb on "reference groups," and Lewin on "gatekeepers" and group 
decisions. Rural sociologists, such as Ryan and Gross, then came into 
prominence. Among user groups studied within this model have been 
physicians, scientists, farmers, and teachers • 

Shortcomings in research in this model as cited by Havelock in- 
clude: the failure to relate adoption decisions to internal processes 

of attitude change or personality or learning variables; the relative 
lack of study of organizations as such; and, the neglect of information 
How wiLiiin organizations. Still, Havelock ranks this model highest in 
empirical research support . 



The problem-solver model . This model assumes that the user is an 
active seeker of a solution to a problem: he has a need, so he takes 

the initiative in filling the need or solving the problem. While dis- 
semination and utilization of knowledge is only a part of the problem- 
solving (need reduction) process, the entire process is conceived as 
approximating the following steps: 

(£) the need is sensed and articulated; 

(_b) the need is seen as a problem to be solved; 

(O a search for sources of relevant information; 

(^) the identification of potential solutions; 

(£) the converting of solutions to prototype actions; and, 

(f_) the try-out and evaluation of the solution. 

If the solution tried does not satisfy the need, the process starts over 
until the need is fulfilled. 

Havelock reports there is less empirical evidence concerning this 
model, as compared to the social interaction model, but that there has 
been a recent surge of interest in it as seen in the establishment of 
organizations and units devoted to helping find solutions, including 
the work of Benne, Lippi tt. Miles, and Watson. 

In the problem-solver model, the user is both the sensor of the 
need-problem as well as the evaluator of the adequacy of the solution. 
Hence the process does not end with something being ''sold," but instead 
it ends with a satisfactory solution to the problem as seen by the using 
agency. Also the emphasis upon diagnosis of the need may be a safe- 
guard against trial of inappropriate products that are available as the 
"solution." An outside agency may serve as a nondirective consultant 
who helps the user exercise his own problem-solving skill rather than 
imposing a solution upon him. This model exercises the internal resources 
of the user in finding his own solutions, perhaps by adapting an avail- 
able solution to his situation. The user must not only accept the sol- 
ution, but by regarding it as his own internalizes the solution into his 
motivational system. 

The weak points of this model as seen by Havelock are: (^) it puts 

the burden on the user, straining his internal resources, it mini- 

mizes the role of outside agencies and available packaged solutions, and 
(_c) it does not provide a model for massive utilization. 

The linkage model . This is Havelock's attempt to combine the most 
outstanding features of the other three models summarized above. His 
synthesis model indicates: 

(^) a focus on the user as a problem-solver who uses outside 
resources skillfully; 

(^) the user and the outside source "go through" the problem- 
solving process of each other by simulation (this establishes 
a reciprocal relationship, not a one-way relationship); and, 

(jc) government facilitates this reciprocal relationship by 
monitoring, providing introductions, seeing that the R&D 




function is performed, establishing networks of social in- 
fluence, and facilitating problem-solving by the user. 

Havelock also discusses other models of dissemination and utiliza- 
tion which might be derived, but which do not appear discernible in the 
current research literature. For example, the historical dialectic of 
Hegel and Marx may support a model based on conflict between forces 
fostering change and those opposing it. The present turbulence in society 
strongly suggests the possible appropriateness of such a model. Havelock 
also suggests that current efforts to use research for highway safety 
systems grows out of a confrontation between automakers and congressional 
committees. This type of event does not seem to fit either of the four 
models reviewed by Havelock. Such events as efforts to reform universities 
to rebuild cities, and to bring racial equality are seen as requiring a 
more comprehensive model to relate these catalytic events and social 
forces and conflicts into a broad social, historical, and political context 



Comments on the Review 



This writer feels that there are current trends and pressures in 
society which make the present literature on diffusion a fairly poor 
predictor of the diffusion techniques and models which will come about 
in the future. This is a primary reason for describing alternative 
mechanisms for educational change in the previous section. Beyond this, 
however, the writer also feels that both the literature of the past, and 
the review of it by Havelock, implicitly assume the permanence of the 
status quo by (^) failing to suppose that current institutions and cus- 
toms will change, (^) failing to relate the research-development-dif fusion 
model to multiorganizational efforts, and by (O falling to grasp the 
likelihood of more pressure from sources external to education, requiring 
it to change. 

More specifically, the literature summarized in the diffusion models, 
Havelock's review of it, and his suggestion of a new linkage model appear 
to largely overlook the following possibilities: 

1. Changes in teacher training . The literature appears to assume 
the permanency of an outmoded system of teacher training, resulting in 
a permanent conservative stance of teachers toward new ideas and tech- 
niques. Thus teachers, by the "social Interaction" model, must be coaxed, 
cajoled, and persuaded to even consider newer techniques; and, by the 
"problem-solver" model, a few are expected to realize the need for change, 
and to seek outside help in devising the solutions. Both these models 
largely overlook the likelihood that a sizeable development effort may 
be needed; they assume either that the solution can be reached by 
applying research knowledge, or that the needed development has already 
taken place. And of course the research-development-diffusion model is 
regarded largely as an intrusion of outsiders with products to promote, 
whether or not these products are ideal solutions for the problems to 
faced. Almost none of the diffusion literature acknowledges as pos- 
sible, to say nothing of predicting, that improvements in teacher train- 
ing may come about with the result that most teachers are trained to 
appreciate current and predicted Innovations, and to permanently retain 



a forward-looking acceptance of and search for realistic opportunities 
to apply new techniques. This writer is reasonably optimistic chat just 
that kind of teacher training is forthcoming, either through present 
training institutions or through the creation of new arrangements of 
teacher training. If this does indeed come about, future teachers would 
neither have to be influenced , as by the ''social interaction" model, nor 
to be solely responsible for initiating solutions to serious current 
problems. Rather, an entire stance of seeking constant improvement would 
be built into their outlook, and they would be aided by multiorganiza- 
tional agreements to facilitate new teaching approaches. 

2. Use of other kinds of schools . The diffusion literature does 
not reflect the possible use of contractor-conducted education, or the 
use of voucher systems, currently being considered or tried in isolated 
instances. It also takes no note of privately operated innovative 
programs for either early childhood education or for remedial instruc- 
tion, such as has been conducted by Lloyd Homme at Westinghouse Learning 
Corporatiou in Albuquerque. While such special programs are few in 
number now, it is conceivable that increasing numbers of children might 
be so educated in the future. And it is not unusual to find that the 
innovator of such programs is also the manager of the program. If any 
of the models reviewed by Havelock would be descriptive of these programs 
it would be the research-development-diffusion model. When children go 
to such schools by arrangement among organisations (schools and R&D 
agencies), rather than by private arrangements with parents, clearly 
neither the "social interaction" nor "problem-solver" model would be more 
relevant . 



3. Public pressures . The diffusion literature reads as though the 
public will take no hand in forcing educational change, by one means or 
another. Nothing is said of community impact upon plans for educational 
change, nor of peaceful or violent protest or rebellion. But the current 
upheavals will surely bring forces for change not noted in current models 
thus leading as Havelock suggests to a new means for change and hence 
new models. As a case in point, the current emphasis of "accountability" 
suggests new methods, new materials, and especially new measures of edu- 
cational outcomes. Surely, various forms of "conflict" models can be 
expected to appear in the relatively near future. 

4. More emphasis upon multiorganlzations . Much of the diffusion 
literature deals with person-to-person relationships rather than with 
relationships among organizations. The writer would expect to see in 
the future many more voluntary cooperation instances among organizations, 
as distinct from between individuals. Of course, the educational R&D 
laboratories were designed with this in mind, even though they are 
presently in reduced circumstances. In the long run, it is predicted, 
important improvements on a massive scale require consortia for develop- 
ment and diffusion. Thus, adoption would be a part of the original 
agreement among the user cind resource organizations. 

5. Powerful new discoveries . Almost none of the diffusion litera- 
ture lists predictions of important and powerful new techniques expected 
in the future. Therefore, the possibility that new discoveries might be 
diffused and adopted by new means has also been overlooked. Just as 
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educational policy research centers are trying to look Into the future 
to suggest needs for new curricula, so we need to be looking Into the 
future when thinking of future diffusion mechanisms. 

It appears, In summary, that the nature of the diffusion literature 
has been to look back and around, but not much forward. If other kinds 
of planning require looking ahead, so likely does planning for diffusion 
in education. 

In addition to the above features of the diffusion literature, a 
final criticism Is that the research-development-diffusion model has 
not been looked at and reviewed In as broad a context as Is necessary 
to judge its adequacy. However, rather than detailing criticisms here, 
the writer will try in a more positive fashion in the next sections to 
outline a broader view of present research-development-diffusion models. 



Successful Development Prototypes 



As indicated in an earlier section and in the opinion of this 
writer, there is a widespread misconception in the literature regard- 
ing how educational change can best take place. There are several 
facets to this misconception. 

First, there is the idea that research findings on one hand, and 
developmental products and techniques on the other hand, can be effec- 
tively diffused by the same mechanisms* Second, there is the assump- 
tion that educational change is to be brought about by the teacher's 
utilization of research findings. 

Borg (1970) has presented a review of two earlier attempts to 
change education: (£) by local innovation strategy, and (^) by edu- 
cational research strategy. He shows that both have failed to bring 
appreciable or widespread change, and he presents his analysis of the 
reasons for these two failures to bring about improvement. He therefore 
recommends "research-based development," which Involves repeated evalua- 
tion and revision of the product or technique until its design objectives 
have been proven to be met, and before it is delivered in user-ready 
form for widespread use. 

In agreement with Borg, the present writer believes that research 
findings are best used by the professional researcher/developer who 
makes discriminating, expert use of research as a facet of his develop- 
ment, and who, in turn, has the skills in both formative and summative 
evaluation to guide product refinement and testing before the product 
is "disseminated" to others. The research-trained developer would appear 
to be the best user of research information in terms of converting 
research knowledge into instructional products. This is not to say 
that other professionals, such as the writer of textbooks, cannot 
utilize research findings to good effect; but, it is to say that efforts 
to disseminate research findings in raw form to the average classroom 
teacher is an exercise in futility. Thus, it is information about 
products and techniques which should be conveyed to the classroom teacher 
and research findings which should be conveyed to the professional 
researcher or developer. 

As part of a search for guidelines for successful developments and 
implementations, it may be useful to digress momentarily to see whether 
lessons may be learned from the military and industrial training settings 
The following historical review of some prototypes of successful devel- 
opments should lend support to the viewpoints expressed above. 



A Military Prototype 

No official history has been written of the mission and operations 
of the Air Force Personnel and Training Research Center (AFPTRC) • Never- 
theless, work done there during the period of about 1948-1957 accounted 
for some of the pioneering work in new instructional development tech- 
niques, including the following: task analysis, taxonomy of learning 



tasks, simulation, programmed instruction, films, job aids, abbreviated 
reference manuals, and many others (Lumsdaine & Glaser, 1960; Gagne, 

1962; Briggs, 1959), 

As noted earlier, these developments were based on some key devel- 
opmental concepts now more widely acknowledged by educational developers, 
such as: clear definition of training goals or objectives (Mager, 1962); 

classification of learning tasks into types of learning and their con- 
ditions (Gagne, 1970), careful sequencing of instructional units and 
hierarchies of skills (Gagne, 1970; Briggs, 1968a); use of the new media 
and media selection (Briggs, Caiiipeau, Gagne, and May, 1966); and, inte- 
gration of these components into an overall systematic procedure for 
the design of instruction (Briggs, 1970a; Tosti and Ball, 1969) • 

It is noteworthy that moot of the instructional products of the 
Maintenance Laboratory of AFPTRC were in the form of booklets, films, 
simulators, that is, concrete devices and materials whose use required 
a minimum of training of the instructor. The implementation of the 
devices in the classrooms of the training center was a matter requiring 
simply the briefing of the instructors. The products were tested under 
controlled conditions before being recommended for widespread opera- 
tional use. The developers of these products were researchers with a 
”flair” for (and a mission of) development. They were capable of 
utilizing research knowledge, but they actually profited more from taking 
what would now be called "a systems approach." And, very Interestingly, 
after the practical classroom effectiveness of some of these devices 
had been demonstrated, they were later used for basic research studies 
in the psychology of learning. For example, a device called the "Sub- 
ject-Matter Trainer" (Briggs, 1959) after experimental proof-testing in 
a classroom experiment was then used to study prompting/conf irmation 
and self-pacing. 

Here, then, is a prototype of how demonstrated Improvements in 
military technical training were brought about. Also illustrated is 
the fact that development can feed research, just as in other instances 
research may feed development. 



An Industrial Prototype 

Several years ago a contract was arranged between the American 
Institutes for Research (AIR) and the American Telephone and Telegraph 
Company (AT&T). This contract was in response to the need of AT&T for 
a more effective and efficient way to teach first aid techniques to 
employees. The several companies of the Bell System had previously 
developed their own first aid training program, required of all employees. 
The average such program required 10 hours of instruction* AIR contrac- 
ted with AT&T to prepare a multimedia program of 7.5 hours, and to 
improve it through formative evaluation and revision procedures until 
it was more effective than the conventional progr. ms. The objectives 
\stixe based on, and compatible with, the Red Cross First Aid Manual, 
which was also the basis for the existing training programs. 

The procedures employed in this program development have been 



summarized by Markle (1967). Performance testing of samples of the 
target population was conducted for more purposes than to revise first- 
draft materials. Tests were used to determine entry knowledge, to 
achieve adequate congruence between objectives and materials, to 
discover where learning time could be saved, to verify instructional 
sequencing and media effectiveness, to check for omissions in content, 
as well as to improve the effectiveness of the program. 

The entire program was subjected to three empirical tests and 
revisions, and some parts of the program (film sequences, narration, 
and text material and practice ques tions) underwent several additional 
revisions . 

On the final program, employees trained by the new course in 7.5 
man-hours scored 270 on a test over the objectives, as compared to a 
score of 145 by employees having 10 hours of training on conventional 
courses, and a score of 85 for employees having no formal first aid 
training. The standard deviation was much smaller for the experimental 
group; in fact, the lowest-scoring person in the experimental group 
scored higher than the highest-scoring person in the conventionally 
trained group — nonoverlapping distributional 

This unusually successful development cos t $250,000 for the 7 . 5 
hour training program. But when one considers that the packaged (and, 
hence, reproducable and exportable) program could be used for many thou- 
sands of employees each day, the development costs were amortized to 
less than one dollar per trainee, and the 2.5 man-hours saved represented 
an additional savings to the member companies. While future research 
is needed to translate the superior test scores to lives saved and 
shorter recovery times of accident victims, the obvious economy of the 
program is attractive when development costs are considered in light of 
total savings through widespread utilization. 

Even when the solution is not a product, but a process or technique 
to be implemented by the teacher or admlnlstra *or , it is important that 
it can be demonstrated or described in concrete form rather than in 
theoretical terms; that is, that it be a completed process, ready for 
implementation, rather than merely the rudimentary components of a pro- 
cess . 



An Educational Prototype 

Borg (1970) states that whereas local innovation may always be a 
good source of new ideas, it rarely provides more than the seed for a 
potentially successful development. While a simple idea, like using 
colored chalk, may spread from a local usage, something as complex as 
team teaching, he indicates, in spite of many successful trials, is not 
in itself a comprehensive program that a principal can visualize and 
implement from the scant information available to him. Many local efforts 
are abandoned for lack of sufficient information from the literature, 
or even from visiting and observing, or attending workshops, oo, most 
local innovative developments, Borg says, are never used successfully 
by anyone except the originator himself. Poor exportability and lack 
of rigorous evaluation are cited as tvo reasons for the failure of 
this method of diffusion, as well as its inefficiency and expense. 
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As an example of a successful educational prototype of ''research- 
based development/' Borg (1970) cites the minicourses developed by the 
Far West Laboratory for Educational Research and Development. These 
are self-contained instructional packages for inservice teacher 
training. A cycle of evaluations and revisions, averaging 18 months 
and 7,000 man-hours at a cost of $100,000, results in a program of 
known effectiveness. But when widely adopted, more than the cost of 
the programs can ultimately be returned to the treasury. Using this 
method relieves teachers of the impossible demand to be Inventors as 
well as practitioners. In support, Borg cites the medical model, in 
which the physician is to diag;nose and prescribe, but not necessarily 
Invent and evaluate new medicines. Highly motivated teachers with 
both expectations are doomed to disillusionment, frustration, and 
failure, he says. 



Educational Development and Developers 



It appears useful at this point to assume that the most widely 
adopted educational developments, for the most part, do and must take 
pi ace initially outside the classroom. Who does this? What techniques 
are employed? l^at kinds of development are involved? 

To take the last question first, most exportable developments of 
wide potential use are products — instructional materials in some form 
or media, rather than theories or techniques. In the example cited 
immediately above from Borg (1970), the products are microfilms plus 
explanatory material, even thouph the purpose of the product is to 
implement a technique, microteaching. In the military and industrial 
prototypes cited, the same was the cas^. An Instructional pac^.age, in 
whatever form, appears inherently more readily usable than does a 
concept or technique, such as individualized instruction. Thus while 
teachers and administrators may profit from information and data on 
new instructional techniques or on methods of planning and decision- 
making, it may require considerable formal training or inservice 
training for effective results. In contrast, many packcged and tested 
instructional products require a minimum of new training on the part 
of the user and, therefore, are readily exportable. 

Now turning to the first question above, those who are currently 
involved with educational development are represented in three major 
areas: commercial publishers, sponsored curriculum development pro- 

jects, and professional researcher developers. The techniques and 
procedures employed by each of those three sources are examined in the 
subsections which follow • 



Commercial Publishers 



Textbook publishers traditionally have supplied the bulk of in- 
structional materials and resources for the classroom. Their model is 
essentially a market model — to produce products that will sell. While 
authors and editors may coninunicate with the users as to kinds of mate- 
rials desired, basically the market determines what will be produced. 

In this sense, the materials sold implicitly determine the educational 
objectives of the courses in which the products are used. It would 
take an independent and enlightened teacher, indeed, to set his own 
objectives and use the products co attain them, rather than permitting 
the content of the products to set the (unspoken) objectives. Textbooks 
are often evaluated more on the basis of the accuracy of the informa- 
tion contained in them than upon the teaching effectiveness of the book. 

Further, most textbooks are marketed without any prior assessment 
of their teaching value or instructional features. True, the author 
may have **tried** the text in draft form in his ofwn classroom, but with- 
out any rigorous empirical basis for evaluating the text. Diffusion 
or adoption of textbooks is typically based on the users* attitudes 
toward the content and format rather than upon its matching vlth 
explicit objectives or its teaching features. 



Apart from textbooks » commercial producers also supply workbooks , 
films » and many of the newer media materials. But in spite of the pub- 
lication of the joint guidelines (DAVI - ASL) suggesting the provision 
of teaching effectiveness datai few commercial products -'^including 
programmed instruction — have undergone the recommended formative eval- 
uation and revision process. 

It is predicted that proof-testing and the use of objectives and 
self- instructional and feedback techniques will not come into wide- 
spread use in commercial products until the purchasers refuse to buy 
products without these features. Then commercial .producers could 
expect to reduce the redundant proliferation of materials in favor 
of fewer » high quality » saleable products because of a demand on the 
part of sophisticated users. If and when that day comes, a n^.w rela- 
tionship would be possible between education and Industry, whereby 
educators set the objectives for materials to be produced to specified 
acceptability levels, with Industry doing the development against 
prescribed standards. Then products which meet the standards would 
be purchased. The mechanism for doing this has been discussed elsewhere 
(Briggs, et al., 1966; Briggs, 1970b). 



Curriculum Development Projects 

In recent years, the availability of federal and private grants for 
massive curriculum development projects has resulted in a number of in- 
novations in both the content and the methods for instruction. It is 
not possible to offer an evaluation of these efforts, but it is relevant 
here to comment upon the development procedures employed. 

In general, such projects fall somewhere in between the methods 
of commercial producers and the methods of the three prototype programs 
described earlier in this paper. That is to say, most such projects 
have not followed a "systems** approach, as now widely accepted by many 
writers on development systems. See especially, Tostl and Ball (1969) 
and Briggs (1970a). 

Specifically, these development efforts differ in how and when two 
crucial elements of the development are considered: (£) the objectives 

of the curriculum, and (^) evaluation of the program in terms of the 
objectives . 

In one well-known science curriculum project it was judged best to 
formulate only the major philosophy or approach at the outset, leaving 
the materials writers maximum freedom to develop materials to encourage 
discovery and inquiry on the part of the student. Specific objectives 
were left to be developed much later in the program. While this^'p?^- 
cedure no doubt did enable the project writers much latitude in cre^Live 
production, ic posed an almost Insurmountable problem when it was timia 
to plan for the evaluation measures. 

Tht-se tv^ elements of development represent the crux of a genuine 
division of opinion among developers. This division finds expression 
most clearly in two opposing views of the merit of stating specific 
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educational objectives in operational (behavioral) terms. While almost 
any ueveloper stands jomewhere on a continuum in this disagreement, the 
focus of the opposing views is shown in the sLatemen'‘.s of various writers. 
Prominent in advocatil.ig specific behavioral objective^' developed early 
in a program is Mager (19o2) . Equally vocal opposition gives a warning 
against too narrov^ ^ treatment of educational objectives. The pros and 
cons of this division are becoming more widely understood and will not 
be reviewed here . 

There is a way to take advantage of the merits of both opnosing 
views. The key to this lies in the distinctions made by Berlak (1970) 
who points out that one may classify the actual outcomes of an educational 
program as (^) intended , (y anticipated , (O unanticipated . This sug- 
gests that one could (a) state the intende d objectives early in the 
development, and plan evaluations of there; (y then list possible other 
( anticipated ) outcomes and plan to evaluate or watch for these; and, 

(O then, after the program is in classroom use, look for any unantici - 
pated outcomes. Armed with results and observations on all three types 
of outcomes, one could make decisions concerning the continuation, modi- 
fication, discontinuation, or adaptation of the program. These pro- 
cedures are incorporated in the writer's suggested ^'systems” model for 
the design of instruction (Briggs, 1970a). 

A second curriculum development project may be contrasted to the 
one mentioned earlier. In Science — A Process Approach (AAAS , 1967), the 
objectives and the evaluation plan were developed early in the project 
cycle. V^ile the fundamental assumptions or philosophical approach are 
in themstiives controversial, being criticized by some (Atkin, 1968) and 
supported by others (Gagne, 1968), this project does contrast with the 
other one cited in respect to the specification of objectives and evalu- 
ation measures. Careful evaluations of student performance, both during 
instruction and at longer intervals, will provide concrete data in terms 
of attainment of the stated objectives, as well as feedback from students, 
teachers, and administrators. Thus, this project has avoided the uncer- 
tainties encountered wiiea these two components are not planned jointly 
and early in the develcc Jient cycle. This project illustrates the devel- 
opment feature of congruence among objectives, meajures, and materials, 
a recommended aspect of a systems approach (Briggs, 1970a). 

Apart from the differences am.^ng curriculum development projects 
in methods of development procedure, as discussed above, the procedures 
for production and diffusion of the products, and the nature of the pro- 
ducts, may often be quite similar among projects. Typically, at some 
point after first classroom tryouts and revision, usually increasing 
in spiral fashion to Include a greater number of tryout schools, the 
project deve** ^ent phase is considered completed. Commercial producers 
usually then in in to reproduce and market the products, which may 
consist of texts and workbooks for students, manuals for teachers, labo- 
ratory materials, films, sound tapes, or other media. From this point 
on, even though evaluative data may continue to be gathered, the adop- 
tion of the program rests primarily upon marketing strategy, subjective 
acceptability of the materials to users, and in some cases, the extent 
of special training opportunities or observations for teachers and admin- 
istrators . 
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Vfhile many millions of dollars have been spent to support the mas- 
sive effort required for such large curriculum development projects, 
often covering many grade levels, there is little basis for sayinp how 
much improvement has been brought about. In the opinion of Watson (1968), 
educators deserve evidence in place of advertising and selecleri anec- 
dotes. He calls for more Viard-nosed information about what happened to 
studeni.a exposed to the new courses — information about what students 
p.ct *alLy learned, what effect the personality and competence of teachers 
had on the results, and "what modes of classroom procedures led to what 
changes in pupils of various types" [p. 16]. suggests that educators 

should not continue to be asked to accept the new courses because they 
were designed by "experts." He expresses surprise that, of all teachers, 
science teachers do not exhibit more of a "show me" demand for empirical 
evidence . 

Another somewhat disconcerting fact is the lack of agreement among 
"experts" as to what should be taught, in whatever subject area (science, 
mathematics, etc.). Whe:; wide disagreement does exist, it suggests a need 
for a brofc.der base of public involvement in the process, although the 
mechanisms for implementing this have yet to be worked out. It also 
suggests the possible need for "problems-oriented" curricula rather 
than "discipline-oriented" curricula, at least for some phases of pub- 
lic education. Work by model cities projects seems to move somewhat in 
these directions, planning a broader basis for the determination of the 
goals of the school system. 



Professional Researchers /Developers 

This broad category of workers may be found operating in a variety 
of institutions and organizations, including universities, private R&D 
firms, R&D centers and laboratories, and military, industrial, and 
governmental research and development agencies. 

Many current professional researchers/developers share a common 
heritage — initial loyalty (by virture of their training) to some discipline, 
such^ as psychology, engineering, sociology, or education — and later 
idenl^ification with a "research-based development" strategy for educational 
change. This identification is commonly brought about by interim experience 
in ol:^^anizations with a mission of development, such as those involved 
with th^ three prototype developmental programs discussed earlier. 

These persons come to implement some development model which empha- 
sizes primarily the development of instructional materials and strate- 
gy ^s, often of a self-contained and largely self-instructional nature. 

Tl ey often begin their developmental interest as an incidental part of 
their research in programmed instruction or other of the new media. 

They also often begin by seeking to relate research findings to the devel- 
opment of techniques for effective instruction. In this respect, B. F. 
Skinner and programmed instruction have had an influence upon educational 
I'cveiui m‘^nts far beyond the specific applications of programmed instruc- 
iJon as a particular educational technique. Practitioners of programmed 
instruction realized (during the period of about 1955 to 1965) that the 



fimpirical testing and revision of programmed texts wa? a technique appli- 
cable to other educational media, leading to a redefii* Ition of '^program- 
ming** as the applicc^tion of a particular set of systematic steps to the 
development of any instructional media oi technique. It is this history 
which probably led to the coining of the term **formative evaluation,'* and 
certainly led the present writer to coin the term "formative design" 
(Briggs, 1970a). 

While many refinements and methodologies remain to be worked out 
in relation to formative and summative evaluation and criterion-refer- 
enced measures, the brilliant work of Skinner in educational application 
was a timely event, arrivi ig in a period growing demand for quality 
education for all and increased accountability for the results of educa- 
tion. It would appear, then, that the future history of education will 
attest to the widespread effects of the work of Skinner and of such per- 
sons as were working in the "research-based development" role during the 
period of roughly 1955-65, and on into the present. The future may also 
make more clear the degree of permanence and realism of the points of 
view embodied in the systems approach, whose proponents so often have 
views conflicting with those of other educators. 

In summary, it appears tp the present vnriter that this rather 
heterogeneous group of research-based developers will, in agreement 
with Borg (1970), p^^oduce more successful educational innovation than 
has been produced by the strategies of local innovation or of educa- 
tional research, per se. In part this may be due to insight into how 
to combine research, development, and evaluation; in part this may be 
due to single-minded dedication to system-oriented developmental models. 
Such an orientation may seem to most classroom teachers and educators 
as somewhat foreign elements in education — a far cry from interest in 
the teacher’s role as a platform performer and heavy dependence upon 
evaluation of input and process rather than output . 
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A Consortium-based Model for Curriculum 
Development and Diffusion 



Believing that past models provide insufficient concepts for bring- 
ing about timely and massive educational changes during periods of press- 
ing public demand for change, and believing that the diffusion literature 
has taken too narrow a view of the potentialities of the research-devel- 
opment-diffusion model, the writer feels obliged to attempt the broad 
outlines of a more suitable model for the future improvement of instruc- 
tion and curricula. 



Assumptions 

The basic assumptions upon which the proposed model rests are as 
follows : 

1. The public will increasingly press for major changes in 
education in the future. 

2. This will lead educational and R&D agencies to form con- 
sortia to bring about agreed upon changes, to be adopted 
by the agreeing parties. 

3. Such consortia may involve a relatively local community 
or larger regions of sizes varying up to several states. 

4. The lay public will be heavily represented in such clusters 
of organizations, and will exert decision-making power. 

5. The goals identified will be based upon a broad review 
of current societal trends and predictions of future 
educational needs. 

6. The goals chosen will require massive new development and 
evaluation efforts, of an Integrated rather than piece- 
meal nature. 

7. Advance commitment to the chosen goals will insure install- 
ation of the resulting developments in the participating 
schools when sufficient evaluative data so justify. 

8. The necessary development effort:? will require new working 
relationships betK-Cen education and industry. While educa- 
tion (including R&D agencies) will take the lead in 
specifying the products needed to reach the selected goals, 
industry will continue to play a major part in the devel- 
opment, production, and distribution, but relating to 
products specified by education (.Briggs, 1970). 

9* As a consequence, adoption and utilization are planned 
and agreed upon in the beginning by all participating 
schools in the consortium. When the results of a number 
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of such consortia efforts are available, other localities 
not having participated in ai'iV consortia are free to 
adapt the products and techniques used in any desired 
consortium effort through cost sharing arrangements for 
amortization of development costs . 

10, The preceding step will require initial support for con-- 
sortia by arrangements among several levels of govern- 
ment and private industry. 

It Is the writer's belief that such a series of events represents 
the best hope of changing education fast enough and responsively enough 
to avoid increasingly chaotic conditions in the public schools. The 
proposed means for change will test the ability of people to work to- 
gether and will require excellent leadership and management. But the 
writer believes that past methods of attempting to effect educational 
change on a widespread scale are not adequate for the future. 



Steps of the Model 

While the general stages of work to be done by such consortia as 
those proposed may vary considerably, the following series of steps is 
proposed as one feasible model, 

1, Setting of educational goals . This is in general the problem 

of looking toward the future in an attempt to foresee the kinds of educa*- 
tion which the adult citizen of the future will need, and working back- 
ward from that to scope and sequences of curricula that would meet the 
need. Doing this, of course, carries with it likelihood of errors in 
estimate of the future, but it also requires serious consideration of 
values* That is to say, some fundamental decision will need to be made 
as to whether the educational goal is to develop citizens capable of 
transforming society, or to help them adjust to some predicted society. 

Few would opt for the transmission of our past culture as the only goal. 
While there are too many philosophical and ethical questions to pursue 
this step further here, coming to grips with such values and goals must 
bo a major starting point. An anticipated major source of va nation 
among consortia is the particular mix of attention to National goals. 

State goals, or more specific regional or community goals. Perhaps the 
first and foremost test of the feasibility of a consortium is the ability 
to reach sufficient consensus upon the goals to be adopted. Certainly, 
new mechanisms are needed for securing the broad base of community and 
other participation that is needed at this stage* A few mechanisms and 
suggestions have been offered by the writer elsewhere (Briggs, 1970b), 

2, Translating goals into objectlver * Once the goals are selected 
for whatever school or school-related systems are participants in the 
consortia, these general goals must be broken down so as to yield year-by- 
year articulated objectives. Such objectives need to be specific enough 
for designing the yearly instruction, or at least by cumulative stages, 

so that at the end of the schooling the goals are met, in so far as 
their evaluation does not refer to postschool, adult life. It is likely 
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that several imtermediate stages of goals and objectives need to be de- 
rived to work downward from ultimate (lifelong) goals to objectives which 
are specific enough to guide the instruction at a particular age or grade 
level. Now that we are in a time for the reform of curriculum objectives, 
it can also be anticipated that there will be a demand for new objectives. 

3. Identifying the enabling objectives . Once the objectives for 
a particular stage or year in the educational process are defined, 
normally it is necessary to study the component parts of each objective. 
These component parts are often called "enabling objectives." Gagne 
(1970) refers to them as "subordinate competencies." Study of such 
component parts helps in planning the instructional strategy for the 
objectives. In many instances, e.g., for hierarchically structured 
component parts, the analysis will help determine the sequencing of 
instruction for the objective as a whole. At least this appears to be 
the case for objectives representing the attainment of intellectual 
skills (Gagne, 1970; Briggs, 1968b, 1970a). 

From this step, onward, writers have disagreed considerably as to 
the developmental steps to be taken. Indeed, there is some disagreement 
as to whether the instruction should be planned by any series of steps. 
However, this matter pertains to differing development models , while in 
this portion of this paper we are attempting to outline an entire devel- 
opment and implementation effort, that is, a full cycle formative evalua- 
tion, tryout, installation, suramative evaluation process. The remainder 
of the steps enumerated here approximate a plan previously described 
(Briggs, 1970a). 

4. Classifying the enabling objectives . In the approach recommended 
here, each subordinate competency (or enabling objective) for cognitive 

or intellectual skill objectives is classified as to type of learning, 
according to Gagne's (1970) taxonomy. (Gagne's taxonomy is preferred 
over others because he also identified the conditions of learning for 
each category.) For affective or psychomotor skills^ other taxonomies 
might be more useful, requiring also perhaps some deviation from the . 
next few steps as they are outlined for cognitive skills. 

5. Identifying the assumed entering competencies . The reason for 
this step appearing late in this sequence is that a very specific basis 
exists for stating entering competencies relevant to the objectives. 
Analysis of an objective as described above leads to identification of 
both the enabling competencies to be learned, and the closely related 
competencies tne intended learners are expected to possess as entering 
competencies. Other more general competencies also may be noted. 

6. Adjusting to actual entering competencies . Any discrepancy 
between the assumed entering competencies and the actual entering com- 
petencies of a given Iramer can then be detected by the development and 
use of pretests over the assumed entering competencies. Deficiencies 

so detected can then be dealt with by any of several means: (£) remedial 

instruction, (y branching programs, (c) reassignment, or (^) modi- 
fication to the learner's program. 
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7* Preparing posttests * For various purposes, posttests should be 
prepared at several levels: (a) at the subordinate competency level 

for diagnostic and remedial purposes; (^) at the level of the objective, 
to monitor pupil progress; and, often, (c) at the unit and end-of-course 
levels for both formative and suramative evaluation of the program. In 
fact, all three ]evels of testing are useful for formative evaluation, 
supplemented by even moi^^ frequent testing in the case oL progranuned 
instruction. 

8. Listing the conditions of learning for each subordinate compe - 
tency" Gagne 's (1970) treatment lists conditions of learning desired 
for all instruction as well as those needed for each type of learning. 

The age and sophistication of the learner may determine how, when, and 
how often to employ each condition. Both the general and the special 
conditions are then consolidated into one teaching sequence, which the 
present writer refers to as the instructional events to be provided. 

9. Choosing between group vs, individualized instruction . There 
are many considerations which may go into the making of this decision. 
Often various competencies or objectives may be assigned to the two 
different kinds of instruction. At any rate, these decisions affect 
the selection of the media to be employed. 

10. Selecting media . Different developers select media at differ- 
ent degrees between microchoices and macrochoices. That is, different 
developers choose the media of instruction for different sizes of "chunks. 
The preferred size of "chunk" could also change the point in the design 
stage at which the choice is made. While further research is needed on 
this matter, the writer prefers the "microchoice," that is, the selec- 
tion of media for specific Instructlcnal events for each competency of 

an objective. The details of how this may be done are elaborated else- 
where (Briggs , 1970a) . 

11. Writing the prescriptions . This is a critical step, especially 
in a large team effort. No matter how closely the design has followed 
the above steps, or how far it has departed from them, it is at this 
point that the educational designers communicate to the des^elopers just 
what items are to be produced, in which media, to what specifications. 

Some of the kinds of details which might go into these prescriptions 
are found in the reference cited in the previous step above. 

12. Developing first draft materials . This may alsj involve a 
large team effort. In any case, it is here that the "prescriptions" are* 
"filled." This consists of a first design of all materials in all media 
specified in the prescriptions. 

13. Conducting formative evaluations . Either brief portions of 
materials or entire "course" segments are tried out with appropriate 
individuals or groups. The assumed prerequisites must be assured where 
tryouts of smaller segments in mid-course are made. Tests are adminis- 
tered at the various levels discussed earlier, and cycles of revision 
and retesting are conducted until the design criteria are met. Those 
criteria are a part of the detailed statement of the objectives (Step 2). 
When design criteria are not invariably uiet after successive revision, 
decisions need to be made about the economy of further revision versus 
changing of standards or ;ther alternatives. 
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14 « Providing amnmatlve evaluatio n > Often this would not be don* 
as a single step, but in the form of a limited Field test, then more 
extensive tryouts, then decisions as to acceptability for ^’operational 
cutover” or ’’institutionalization.” 

15. Monjj^r.^ng and diffusion . As said earlier, assuming that the 
evaluations indicate that design criteria were met under field test 
(normal school) conditions, it Is assumed in this model that all cooper- 
ating schools adopt the program. In any case, continuous monitoring is 
desired. In this sense, diffusion among participating schools in the j 
consortium was assumed at the outset. Previous comments were made as 

to how diffusion beyond the consortium might come about. \ 

16. Training of teachers . The role of teacher training has been 
left as a late entry here because of earlier predictions made as to the 
future improvement of teacher training. Were this not assumed, teacher 
training would come at an earlier point. In the ideal consortium of 
the future, the teachers would have been trained in current and expected 
new techniques during preservice training, and so might need a minimum 

of special training as the program develops. Under less ideal conditions, 
the inservice training would begin early and continue through orienta- 
tion, observation , discussion , etc. 

17. Permanent reassessing of needs . In the ideal consortium, the 
group effort would not cease with implementation of the program. A small 
team would continuously scan the horizon of the future, comparing these 
observations with the nature of the ongoing program. By such contin- 
uous monitoring of the program and forward search, the consortium might 
avoid the need for such a major overhaul as that just completed. Contin- 
uous adjustments should replace future major problems, upheavals, and 
need for emergency measures. 



Discussion of the Model 



Many secondary considerations have been omitted from this descrip- 
tion of the model. For example, one reason tor continuously monitoring 
the program is to detect unexpected or unplanned outcomes of the program. 
In fact, even at an earlier stage, detection of ’’emergent or expressive 
objectives,” as distinct from the ’’planned objectives" is a desirable 
function of evaluation efforts. 

While some details of the recomended development and diffusion 
model would need to be changed for affective or psychomotor objectives, 
the writer would argue that the validity of the overall systems model 
is as likely for these as is its validity for cognitive objectives. 
However, fewer examples now are available of its application to the 
former domains. 

While specific attention is not given in the development and 
;iFfuslon model to effective, emergent, or unanticipated outcomes of 
.nstructlon, the utilization of formative and summative evaluations 
makes the model at least potentially sensitive to such outcomes. 
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It is further recognized that the model presented does not obvLously 
espouse "humanistic" approaches to curriculum design. But there is no 
reason to suppi^sc that a system-oriented, consortium-conducted model 
for the developm.ent and diffusion of curricula cannot be humane. On 
the contrary, the recommended broad base of societal input to consider 
the desired values and goals goes far toward building humanism into the 
system. Once the system is in operation, a single tyrant in charge 
of a classroom might be more easily detected. 

I 

As an example of a "systems" type of Approach, the suggested model 
may be fairly representative of other "systems" models as presented by 
other writers, especially at the opening and'! closing steps outlined. 

But in Steps 3 through 10, the details of the model as given here may 
not be representative of other systems models. These steps involve the 
issues of how course structure and sequence are handled, how instruc- 
tional strategies are developed, and whether media are chosen at the 
microlevel or the macrolevel. 

It may also be observed that the model presented here assumes a 
large team effort, using more .sophisticated and specialized personnel 
than would be available in some more limited development efforts. It 
also assumes the probable validity of varying instructional strategies 
in accordance with different types of learning, as reflected in Gagne’s 
(1970) taxonomy. 

The model also focuses upon preparation of predesigned instruc- 
tional materials as a key resource; it assumes extensive use of self- 
instructional materials and individualized methods of Instruction. It 
presupposes that some day the Inservlce training of teachers will 
lead teachers eventually to welcome such resources, and to develop 
teacher roles compatible with such tools: for learning and Instruction. 

Since it focuses upon the above assumptions, the development model 
presented here does not say much about the learning environment and 
context, or cost-effectiveness, or roles of people. Under the consortia 
arrangement proposed, both management models and context models would be 
dove-tailed with the development and diffusion of curricula. And cost- 
effectiveness cannot be compared to present education, whose effective- 
ness is mostly unknown. Where there are no objectives, almost any eval- 
uation is suspect. But there is precedent for asserting that the large 
development costs of a systems-based development and diffusion model 
often amortize out in a highly satisfactory way. In any event, the 
model, emphasizing objectives and evaluation, lends itself to determin- 
ation of both cost and effectiveness. 



Some Elements of Success for the Model 



Whether one agrees with either the underlying assumpti^ as of the 
recommended model, or with the model itself, the question of its 
feasibility is a necessary one to caise. Apart from the obvious coopera- 
tive and Joint actions among many kinds of organizations, society, and 
government, what else would it take to make it work? Several possible 
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elements for success^ othei «nan those already discussed suggest them- 
selves : 

1. More face-to-face contact among innovators, teachers, 
teachers in training, industry, school, and society. 

2. More planned mobility and communication, visits, demon- 
strations, and other interchanges so that the basic out- 
liuKs of the model and possible innovative changes are 
made aware to all these groups . 

3. Much more use of visits, demonstrations, team teaching*, 
visiting teachers, and the like for learners, teacher 
trainees , teachers , researchers , developers , and all con- 
cerned in a consortium effort. 

4. A better mix of formal, on-the-job, and real-world learning 
experiences for teachers , professors, and other personnel . 

5. Open critique of the consortium effort as to maiiagement , 
organization, procedures, and results, so that the entire 
concept could be changed to make it more effective. 

6. Much communication among member institutions of the con- 
sortium, policy research centers , government agencies, 

£ind the public, not only in the planning phases, but 
permanently. 

7. Continuous study of the child, the school, and the society, 
with the view of reaching new insights to improve future 
practices • 

8. A reappraisal of the function of the university in society, 
with special regard for the optimum ways to accomplish 
research, development, teacher training, evaluation, and 
diffusion* 

9. Introduction of trainees in the RDD6(E process, and constant 
trial and evaluation of how to coordinate their training 

in a meaningful and efficient context. 

10. Continuous rethinking of the research, development, diffusion, 
and evaluation roles to be played in the consortia. The 
trainin, of RDD&E specialists offers many challenges as to 
needed future function and context of function, vniile study 
of the RDD&E functions in the present contexts in which this 
work takes ^x^ce is a necessary effort, the matter of plan- 
ning the future training of these specialists requires the 
same looking to the future as recommended here for development 
and diffusion of effective procedures and products for all 
education. 
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Organizing for Change 



One of the Implications of the recommended model Is the need for 
careful attention to organizational structure, management, and leader- 
ship. In the proposed model, diffusion is by organizations rather than 
by individuals. This very fact does indeed call for methods of control 
and the development of flexible management structures, policies, and 
techniques to coordinate the work of many persons. Better than usual 
coordination among organizations will be required. 

It is beyond the purpose of this paper to suggest how educational 
institutions need to chrnge. Implicitly, however, we have suggested 
thrit some educational changes need to involve either new inrtitutions 
or new cooperative arrangements among present institutions, or both. 

No doubt other needed changes are changes in goals, bringing about a 
need for new curricula, and new methods and materials, '^e have also 
suggested that we believe the public wixl require changes, or will create 
new institutions as an alternative. 

Mechanisms for more meaningful ways to e.rrive at goals are greatly 
needed, as well as ways to progress from goals to objectives. For 
too long the lofty goals of education have remained imrelated to the 
curricula. 

It is not at all clear to the writer whether the future changes 
ia education will consist more of changes in organizations, or in goals, 
or in cur'i Icula, or in methods and media of instruction. No single 
breakthrough in any of these areas seems to suggest the future direction. 

The proposed consortium-based model for curriculum development and 
diffusion does provide a systematic method for monitoring the con- 
gruence of the educational methods and evaluations with the objectives. 
Unless these three major components of the process are compatible with 
each other, no evaluation and, hence, adaptation is possible. In the 
words of Mager (1968), we need to ask: "Where am I going: how shall 
I get there; and, how will I know I've arrived?" (p. 7). 
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A CRITIQUE OF THE PAPER 



Norman J . Boyan 

University of California — Santa Barbara 
Author’s Aims and Intentions 



Professor Briggs’ primary aim is ”to present a View of development 
and diffusion activities that will assist in making then 

more effective for the improvement of education." (p. 144, italics added). 
His secondary aim is to describe selected considerations that prompt 
him to advocate his preferred "model^* of cducat ional development and 
diffusion activities. Hir: Intention is to offer "a selective and 
illustrative view of the purposes and means of educational development 
and diffusion as they have taken place in the past, as they are currently 
practiced, and as they might occur in the future." (p. 145, Italics 
added). He disclaims any intention to produce a comprehensive review of 
the state-of-the-art in educational development and diffusion. 



Summary of Critique 



The author does achieve, in part, his primary aim. His advocacy 
of a systems-or iented, consortium-based "model’* Incorporates his view 
of how to make curriculum development and diffusion more effective. 
However, he offers only a bare outline of a "model," not a working 
drawing . 

As for his supporting aim, he does, indeed, present selected con- 
siderations that compel him to propose the consortium approach, but his 
presentation is so selective that the reader who expects a detailed 
topographic map of both educational development and educational diffusion 
comes away disappointed. He finds, instead, an outline map of the two 
domains, without sharply defined relationships between constituent 
element! and without clear indication of the four compass points. In 
addition, the reader who seeks a strong conceptual bridge between the 
pea.es of educational de\*elopment and educational diffusion comes away 
dissatisfied. He finds, instead, a set of building materials, without 
clear assembly instructions and without assurance that the assembled 
pieces will reach from one peak to the other. 

The paper suffers from a number of substantive deficiencies, weak 
organization, *and uneven presentation. Overall, the major idea of the 
consortium-based approach commands more attention than the arguments 
advanced in its behalf or the rudiments of proposed structure and 
operation of the "model." 

Directors of the Oregon Studies project must share responsibility 
Cor the paper’s modest success. They commissioned too large a task. 

Tiiey would have pursued a wiser course by seeking two separate papers. 
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one on development and one on diffusion, allowing the authors to treat 
the boundary lines as permeable where appropriate to support arguments 
and views of how these domains do or should relate to each other. 

The author implicitly recognized the enorr*ity of his assigned 
task when he presented his statement of what he in^“^ ulcd and what he 
did not Intend to do. Ho deserves credit for the legit imatc attempt 
to narrow the scope of his paper and for his partial achievements. 
Still, the paper's shortcomings diminish the value of its positive 
contributions. 



Analysis of Major Thesis and Support in g Arguments 



Professor Briggs* essential thesis is that a consortium-based, 
systems-or lented , research-based development 'Wdel** stands superior 
to other approaches for producing and installing the "timely and 
massive** improvements required in education today and tomorrow. The 
twin emphases on high quality development work per sc and on need for 
effective change in educational practice dominate the author's arguments. 
His stress on the critical nature of the linkage between development and 
diffusion deserves high commendation. 

Identification of the main lines of argument, as presented below, 
require the reader to consolidate, at times, points appearing in several 
sections of the paper. In the analysis of arguments, the voice of Pro- 
fessor Briggs will be identified when he is speaking; otherwise the 
critjquer's voice will be heard. 

Compelling N e ed and Great Public Demand for Change 

Great public demand and compelling need for "timely and massive 
char ,^c** in education appear as givens in a series of badly assertive 
statements scattered over the paper. Commonly available opinion, in- 
formation, and hard data offer support for the "compelling need" posi- 
tion. Despite the author's omission of specifics, little basis exists 
for criticizing his categorical stance on the need for change. Indeed, 
he could well have stipulated the need-f or-change as a given or a basic 
assumption in the very beginning and left It at that. The additional 
references to the need serve no constructive purpose. 

The author sets great public demand for change as the other side 
of the compelling-need as sump t ion. The relevance to his thesis o f 
the public demand issue falls into two categories. First, ht: says, 
the means now available cannot deliver the kind of change (liL^ely, 
massive, integrated, responsive) the public really desires. In his 
view, the public's readiness to consider the establishment of new 
institutions for education and schooling, its press for quality 
education for all, and its emphasis on accountability show how deter- 
mined it is to secure the changes which will fill its wants. If 
presently available means are inadequate, new means must be created; 
for example, a consortium-based arrangement for curriculum developuent 
and diffusion. 
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Second, continues the author, "the public" does and should serve 
as the legitimate source of goal definition for education. Lay In- 
volvement desirably encompasses reviews of current social trends and 
alternative educational futures, presumably generated by specialists. 
However, the public will and should exert decision-making power. Tlu* 
proposed consortia attend properly to this requirement. 

The author does nat make clear whether the public declslon-makie- 
power will be exercised directly or representatively In goal-setting; 
whether public power In a consortium will be exclusive or participa- 
tory. Since details never appear on exactly how tlie public w ll 
participate or be accommodated in the proposed consortium arrangements, 
the contribution of the argument on public involvement to the ack-hor's 
central thesis falls short of any useful mark. 

Moreover, his notion that public participation may helpfully con- 
tribute to reducing divergence among experts on what should be taught, 
and by Implication on the goals of education. Is naive. The messages 
o^ students of the politics of education unequivocally point to absence 
of consensus among lay citizens as one of the root causes of lack of 
ability to state educational goals with any precision and to reach fun- 
damental agreements thereon. These students tell us that the public 
does not exist . Rather there are several or many groups who come to 
the declslon-maklng table with different desires, wants. Intentions, 
and preferred practices as well as goals. The author recognizes that 
the resolution of these differences is crucial, but offers no indication 
of !iow his consortia would promote the needed resolution or achieve 
consensus on goals. 

The proposed division of labor between "public" goal-setting and 
"professional" objective-setting Is an Important Ingredient in the 
author's thesis. For it Is, in part, the public's direct legitimation 
of goals and indirect legitimation of objectives that undergird the 
expected advance guarantee by practitioners to adopt products generated 
by developers, when the products meet preset design specifications. 

The agreement-ln-advance to adopt products serves as the main pillar 
of the diffusion capability of the proposed consortia. 

Unfortunately, the au*:hor Ignores the fragile nature of the govern- 
ment and governance of schools and other educational agencies and the 
hazard of treating advance agreements as very stable outside of a 
specific contractual arrangement. His advocacy of a consjrtium as an 
instrument of diffusion appears, therefore, to rest on a rather frail 
reed, unless he meant to incorporate a contractual ba^ae into the con- 
sortium's practices. He might have dispelled the sense of frailty if 
he had drawn his picture of consortium structure and operations in more 



The author briefly examJ the following mechanisms; the aroused 
society, the concerned community, the local classroom teacher, the re- 
searcher, Industry, and a combination of educational and research insti- 
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Consideration of "Mecl.aulsms' the Process of Change 




liitiorib Coming together In a consortium which incorporates the public, 
Industry, and possibly state and federal agencies. The cursory treat- 
ment of the first five serves as a foil for the conclusion that a new 
device Is needed which a 1 lows the publ ic to set genera 1 goa 1 s , prof es- 
slonal educators to translate the goals into derivative product speci- 
fications, and industry to produce the desired products. The only al- 
ternative mechanism considered is the proposed consortium. 

The cryptic treatment of the inadequacies of the five other 
mechanisms brings the reader to his first meeting with the consortium 
In a hostile frame of mind. The summary table only exacerbates the 
hostility. If the author had incorporated the references to the 
advantages of the consortium approach Into his later discussion of 
Ills preferred model, there would have been no reason to include this 
section in the paper. 

His brief loo^ at the process of change serves approximately the 
same purpose as the treatment of mechanisms; namely, a foil to make 
a point and an opportunity to make some foreshadowing remarks. Both 
the point and the remarks prove to be redundant. 

Lack of Future Orientation of Literature on Diffusion 



The comprehensiveness of the material Included in the volumes re- 
cently published by Havelock and his colleagues lends validity to the 
author's stance of treating these reports as a proxy for all literature 
on diffusion. His use of the Havelock reports leads him to conclude that 
the existing literature looks back, around, and not much ahead; it is not 
enough future-oriented; it serves as a poor predictor of what educational 
diffusion could and should be; it Implicitly assumes permanence of 
the status quo. 

The author cites five specific "possibilities** grievously overlooked 
by extant literature as potential sources of future changes. These 
"possibilities'* Include potential changes in teacher preparation to 
equip practitioners to be more responsive to innovation, the likelihood 
of strongly Increased public pressure for rapid and massive change, the 
promise of multlorganlzatlonal arrangements to produce and to install 
desirable change, the emergence of new kinds of schools, and powerful 
new discoveries. The value of the citation resides more in its utility 
for exposing additional dimensions of the prepared consortium "model" 
than in its substantive power. 

Inadequate Treatment In the Diffusion Literature of the 
Resear h-Development-Dlf fusion Model 

The gist cf the author's argument here is that the scholars 
and students of diffusion have poorly represented the optimal linkages 
between research and development on the one hand and between development 
and diffusion on the other* Unfortunately, the categorical assertions 
that Havelock, ^. , standing as the proxy, differ In their view of 
the research-development-diffusion model from the author do \ot receive 
sufficient documentation to make the difference explicitly clear. The 
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equally catefoii:al aaaertion that Havelock’s reports neglect some impor- 
tant features of the research and development process and Its connection 
to diffusion alf stands weakly supported. 

The only siocific criticiam offered of Havelock’s treatment is Ills 
overfinphasis jn tlic linearity of process and direction of flow In tlie 
restarcli-(levrlo}7Tncnt-d if fusion model. Even this rritlcimn comes after 
acknowledgment that Havelock, • do note the importance of feed- 

ha--k and redesign. An Impl le d criticistn appears In the discussion of 
the "possibility" of changes in teacher training to the effect that "the 
rcscarch-developncnt-dif fusion model is regarded (in the diffusion 
literature) largely as an intrusion of outsiders with products to pro- 
mu whether or not these produces are ideal solutions for the problems 
to be faced" (p, 156). By implication, the authcr Is more concerned with 
Hi U ^ 'ons than with intrusions. Another implied criticism is that 
Btu cots of diffusion overemphasize the adoption and spread of innova- 
tions whereas the author weighs development per se more heavily in 
considering the several components of the research-dcvclopmcnt-dif fusion 
model . 

In brief, the author leaves the reader on his own to compare and 
contrast the Havelock, ^ al^* # treatment and the Brigg's conceptualization. 
This is a weak and cavalier way through which to support a main line 
argument for the thesis. 

The superiority of tested products over the utilization of knowledge 
as a direct source of change represents one of the major differences in 
perspective between the author's view of the world and the view commonly 
expressed in the diffusion literature. The proposition enters early and 
receives continuous stress. It is central to the over-all position adopted 
and advocated by the author, along with his concern for introducing well- 
developed products into effective classroom use on a widespread s/iale. He 
sees "inherent" advantages in products, especially those which come in 
the form of complete Instructional packages, because of minimum require- 
ments for additional training of intended users. By implication, adopt- 
ion and exportability are Inversely proportional to the airount of add- 
itional training required . 

The power of instructional products as an Instrument of diffusion 
and of improvement of education, as compared to the use of other strate- 
gies and tactics, is an appealing argument to almost anyone who has come 
close to rigorous development work In recent years. The appeal of the 
argument Itself, however, does not compensate for omission in the paper 
of documented evidence to support the basic proposition. The author 
simply does not parade sufficient support for his case in his citation 
of the three prototypes and in his description of the work of commercial 
publishers, curriculum-development projects, and professional researcher- 
developers. 

R esearch-based Development Strategy vs. Local Invention and lunovatlon 

Another building block in the author's construction of the consor- 
tium-based "model" is the argument that a rigorous research-based develop- 
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ment appror^c)i offers more liope for produclnK nnd Introducing tested im- 
provements over a wide base than an approach which anticipates the inven- 
tion and diffusion of meaningful Innovation by classroom teachers. 

Again, the line of argument appears early and €?mcrges often. Sup- 
porting assertions imlude the following! the innovations developed by 
teachers are not likely to move Inti) widespread use because there is no 
handy organization to diffuse them; the dissemination efforts associated 
with local teacher-made Innov.it Ions , llku? those of r cscarclicrB , have not 
proved spectacularly successful; "precious fev improvements" come from 
efforts to persuade teachers to try to adopt procedures which are ap- 
plications of research findings; "cajoling, pleading, coaxing, and 
explaining" do not work because the efforts are too far out of context 
for; most teacheri- to make the application; engineering by professional 
reBcarcher-developers is needed more than translation; dependence on 
Isolated practitioners of diffusion (change agents) will not generate 
widespread and significant change; it is a tnlsconcept ion to assume that 
teachers' utilization of research findings will lead to educational 
change; attempts to disseminate research findings In raw form to the class- 
room teacher constitute an "exercise In futility"; local innovation may 
be a good source of new ideas but it rarely serves as more than a seed 
for potentially successful development; most local innovations arc never 
successfully used by anyone other than the original developer; the 
low level of cxportabll ity of local Innovations Is associated with the 
characteristics of lack of rigorous evaluation, of Inefficiency, and of 
expense; teachers who aspire to be Inventors as well as practitioners 
"are doomed to disillusionment, frustration, and failure." 

The asserted deficiencies of the local Innovation strategy serve 
to introduce counterpoints, either explicitly or by implication, to 
support the author's choice of the research-based development strategy 
for improving both the production and diffusion of educational improve- 
ment. For example, translators of research findings are less Important 
than "educational engineers who can apply research findings In the design 
of workable (and testable) prod^icts, techniques, tools, or Instructional 
packages" (p, 149). A consortium approach will generate more widespread 
and significant change than Isolated change agents. The technical 
superiority of research-based development stems from Its emphasis on 
continuous formative evaluation and revision of products or techniques 
until design specifications are met. Its strategic superiority stems 
from ability to make delivery In user-ready form on a wide scale. 

The contrastive approach allows the author to diaw the conclusion 
that research-trained developers are likely to be the most productive 
users of research knowledge, which they have the competence to convert 
into instructional products, and that teachers are likely to use most 
productively Informs f ion about products and techniques. The separation 
of functions and division of labor to which the author's line of argu- 
ment addresses itself constitute key foundation stones on which he 
builds his proposed consortium. 

Experienced observers of attempts to introduce worthwhile educa- 
tional Innovation will tend to agree with the view that the "best" users 
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of resecircti loiowledge are professlcnal researcher-developers; certainly, 
ttiey appear to represent a more reasonable user Rrnup than classroom 
feathers. Er:|ual support will cone for the notit.n tlial lotal classroom 
te.acliers are better users of Information about tested products and 
teclinlqucs than of either raw or translated resenrcl' knowledge, hven 
•<o, the reasonableness of the author's position could have been streriKth 
cned If he liad depended less heavily on the views of only one other 
pract Icing educational developer, 

llxamples of I’rototypes of Development Work 

It Is Important for the reader who really wants to understanc' Ri Iggs 
to note clearly the operations and activities which receive praise In 
the cited military, industrial, and educational piototypes. He uses 
the examples primarily to identify the following guidelines for successful 
development work: clear definition of training goals or objectives; 

r lass! f Icatlon of learning tasks Into types of learning and their con- 
ditions; careful sequencing of Instructional -inlts and hierarchies of 
skills; use cf the new media and media selection; integration of the 
components into an overall systematic procedur*' for the design of in- 
struction; minimal new or additional training ^f users; the Importance 
of performance testing for revision of first draft materials, for 
determining requisite entry knowledge, for achieving congruence between 
objectives and materials, for discovering where learning time can be 
saved, for verifying instructional sequencing and media effectiveness, 
for checking omissions In content, and for improving overall program 
effectiveness; the importannee of a continuous cycle of eveluatlon and 
revision. The Industrial and educational examples Indicate the great 
potential for reasonable amortization of high Initial development costs 
through widespread adoption by a large number of users. 

The author also cites and emphasizes the value of development to 
stimulate and feed research, as well as the more expected vice versa; the 
strong dependence of LoDlementation and utilization on the production or 
availability of thoroughly "completed" products or processes; the riOtion 
that adoption and diffusion are inversely related to the amount of 
additional training by the intended users; and the superiority (ot 
diffusion in education of completed products over knowledge in its raw 
form. 



The discussion of the military and industrial prototypes represents 
the most confident and plausible part of the paper. The author has pro- 
vided nicely/ sketched examples; in each case it Is evident that he knows 
the territory. The presentation of che educational prototype suffers 
because of lack of detail and the allocation of most of the space to 
comment on the inadequacies of the local innovation strategy as com- 
pared to the research-based development strategy. 

The presentation of the three prototypes would have psroflted enor- 
mously from a crisp sumruiry of the dimensions of successful development 
vw'rk which the author applauds in each of the examples. It Is Just 
;< Sibil that if the author had taken this step, he would have recognlzeo 
LSe redundancy between much of his argument on development in the proto- 
types section and in other parts of his paper, particularly the so-called 
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of the Model . 

There 1b need also to examine what the author docs not Bay about the 
examples he has used. Each addresses Itself to the tralnlnj^ of peraonnel . 
Training, per se takes precedence In these examples over content or know- 
ledge per se * The basic orientation held by the author may prevail only 
(jr primarily when developers address themselves to the specific training 
of a blown target group rather tlidn the education of more diverse groups. 

A real burden of proof on this score rests with the author. The Issue 
hero Is an authentic one, but receives no systematic consideration, not 
even In that section of the paper on ''Educat lonral Development and Developers** 
where the opportunity to cope with the matter readily presents Itself. 

The Professional ReBearch-nevelopers vs. Commercial Publishers a nd 
Curriculum Development Projects 

The author’s criticisms of the coraroerclal publishers stress two 
points. First, they use a market model of development and diffusion 
that caters to users' attitudes about content and format rather than 
a desired match of products with explicitly defined objectives. Se- 
cond, few commercial products c>VGr undergo an appropriately higi or rig- 
orous enough process of formative evaluation together with continuous 
revision based on achievement of specified objectives. 

Foundation- and government -sponsored curriculum development projects 
deserve credit, notes the author, for the introduction of much innovation 
into education. In both content and method. The primary weakness of 
curriculum development projects, he observes, is that they have not 
sufficiently used a systems approach to achieve a congruence of objectives, 
measures, and materials. 

The *'good guys** are, of course, the professional researcher-“devclopers. 
Explicitly and categorically the author declares that the professional 
researcher-developer group will produce more successful educational in- 
nov.it Ions than any other group. **In part this may Le due to insight into 
how to combine research, development, and evaluation; in part this may be 
due to single-minded *ded icat Ion to system-oriented developmental models. 

Suci an orientation may seem to most classroom teachers and educators 
as somewhat foreign elements in education — a far cry from Interest in 
the teacher's role as a platform performer and heavy dependence upon 
evaluation of input and process rather than output . ** (p. 167) 

The value and contribution of the line of argument are mixed, at 
best. An example or two to document how commercial products do, in fact, 
fall below a desired norm of formative evaluation and revision would 
have been more convincing than the generalization which appears. The 
author's discussion of the genuine division across curriculum development 
projects of the merits of stating objectives in advance and in operational 
terms is a useful contribution. People who have been close to curriculum 
developers stand well aware of the power and virulence of this rift as 
pertinent to funding decisions and reputational status. The view that 
broader public involvement may assist in reducing divergence on educational 
goals and objectives, as already indicated, betrays naivete. The suggestion 
that use of **problems-or iented** curricula rather than **dlsciplinf*-or iented** 
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( tirrlculfl may help to Incrcaat’ - ona»*nBUfl on goals and objectives lacks 
any real meaning because there Is no Indlr-.-it (on of wfiat either looks 
like or of how the differences between them translates operationally 
Into achieving greater agreement. 

Kven If the reader Is inclined »o agiee that the professional 
researcher-developer g“o'ip will ^jodtice more successful (tliat Is, 
appropriately tested and widely adopted) Innovations than other classes 
of developers, the author must stll’. be favlteu because of his tactic, 
of exposing the weaknesses of others while stating only the strengths 
of his preferred group. 

One of the more pertinent aspects of the consideration of major 
classes .f developers receives no consideration. Descriptions of the 
sponsnrcBd curriculum development projects and of the work of professional 
resoarcher-developcrs suggest, but without pointed reference, thst the 
latter tend to be more centrally concerned with Instructional delivery 
wtille the former group tends to co xentrate more heavily on Instructional 
content . Studemts of educational oevelopmcnt know that this distinction 
really exists. They further know that the bridge between the two groups 
remains unfinished; In some places, indeed. Its building has not even 
been considered. The author could profitably have explored desirable 
rapprochement to Indicate how be expects to cope with the distinction in 
his proposed consortia. 

Tlic Consort ta-based Model for Curriculum Development and Diffusio n 

The culuffllnatlng argument for the consort la-based "model" appears 
in thu statement of a set of 10 "Assumptions," 17 "Steps of the Model," 

10 "Elements for Success," along with a brief "Discussion" and a few 
observations on "Organizing for Change." The way in which the author 
chooses to organize and present his case does net provide a neat, crisp, 
and clearly articulated description of structure or operation of the 
"model." Specifically, the "Assumptions" and "Some Elements for Success" 
have more in common with each other and with several of the "Steps" 

.han they are represented. They would have carried more weight together 
tlian they do separately. 

The assumptions really consist of a set of anticipated and desired 
conditions. The author himself uses the following words: ". . .such a 
series of events represents the best hope of changing education fast 
enough and responsively enough to avoid Increasingly chaotic conditions 
in the public schools" (p.l69; Italics added). The simplistic cause- 
effect relationship between the first two "assumption-events" Is definitely 
moot. Exactly why the author believes that strong public press for major 
changes In education "will lead" necessarily to the establishment uf 
consortia, involving educational institutions and R&D agencies, de- 
serves much more considered explanation and justification than ever 
appears. A set of "assumptions" of such strategic importance to the cen- 
tral thesis of the paper can not legitimately be left so barren of 
‘ ' pport . 





The otht*r l(»n-evr»nl ^ supp 1 v inadpquri t r c 1 ups t n Hip ope 

flind opprntin^ cofvJitIcMS of the proposed ronsorti;i. Vnr ex/imple, despite 
the obvious ImpHcntior of n (ontrafturi] oblif^aMon in tti€- f»peraMoii of 
a c onsor I iuin, the atitlior never sp«*( i f iral ly nentinns (ontracis or t lie 
question of what miKht happen if the developers Ctinnot produi e nr the 
operating, agencies decline to adopt. The issue of appeal routes or ad-^ 
Judication receives no attention. 

Tfie **Steps of the ^Jodc^' represent a sitnllar mixed baj? oi 17 state- 
ment s , with brief and close to crypt Ic explanations. Tfie r r vpt i ( feature 
of the **steps** material shows itself In the number of **elsewhere references.** 
On no less than rlx occasions tfie author uses such statements as, '*lhe 
details of how this rway be done are elaborated elsewhere 1970a).'* 

Four of the **steps** (1, 15, 16, 17) really pertain to the operation of 
the (. f^nsor t lum . The other steps pertain to the sequence of op^M /iflons In 
tin* aut!ior*s Ideal Ircd process of development worV . 

Tfie entire paper would have prospered, and the reader would liave 
benefited. If all of the references to the preferred style of development 
work had appeared In one place rather than scattered over the se< t l(ins on 
prototypes, ma^or classes of development workers, and the model itself . 

Had the author brouf^ht all of this mater lul together, he might have more 
successfully and convincingly Indicated why his preferred style 1« 
superior to others. 

Similarly, the paper would have been strorger if the author l).ad 
collected in one place all o! fils major references to the proponf‘f| consor- 
tium .approach. He might have then seen the need to offer a more detailed 
treatment of how a consortium would actually work. 

One area where the author especially owed the reader much grcMtcr 
clarity Is on the relative contributions of the consortium proposal to 
development qua development and to diffusion per se . Tn hJLa own ”ni«- 
cus8l<*n of tfie Model,** he bounces back and forth from onv domain to 
the other. Simply put, the arguments and case presented by the autfior 
really never make clear whether the consortium will contribute primarily 
to the Improvement of development work or primarily to the adoption of 
tested .il ternatives on a widespread scale or equally to both. It can be 
assumed, on the basis of a number of explicit and implicit statements, 
tf)at the author wants to see improvement in both development and diffusion 
in order to enhance their effects on securing Improved practice in the 
scf)ools. The way in which he presents his case, however, tends to leave 
tfie reader with the feeling that tfie consortium approach will really lielp 
diffusion more than development and production. It is liard to hee fiow 
the nondeveloper participants in the consortium will make any real con- 
tribution to development qua development, unless it is in goal-suiting, 
objective specif icat Ion, and sites for field testing. None of these 
dc t Iv i t les really requ 1 ru a consort lum for their presence in devul opmeiif 
work per se . 



<•>)))• iiid IttK MlNlwnit>nt. 



Wien tlie render flnlshe» the paper, he will find he has learned 
notnethlng aboiii educational development, especially If he Is willing to 
work at pulHrig pieces together on his own; something about educational 
diffusion, but less than he might have elsewhere; and not enough about 
what the authoi really means by "a consortium-based model for curriculum 
development and diffusion.” The "model” never appears as a complete 
entity. The paper as a whole Is particularly disappointing because the 
® consortium based on sniltlorganlcatlon arrangements and 
affiliations possesses great appeal for audiences seriously committed 
to Improving both development sod diffusion and, In turn, educational 
practice. 
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Ref fontif' to the Critique 
Leslie J. Brlf./.n 



Sln<!e a pap,c limit has been set for my response to the critique, 1 
cannot possibly reply to all of the critical comments. This Is Just as 
well, since so many of the criticisms made would be useful mainly If a 
new draft of my paper were planned, but such Is not the case. The cri- 
tique Includes many suf^gestlons for more documentation here, less docu- 
ment.atlon there, etc. 

1 am as disappointed In the critique as the critic Is disappointed 
In my paper. I thought the purpose of the critique would be to suggest 
better solutions than I have suggested, rather than to suggest how the 
paper might be rewritten. 

The critic does not suggest a better solution to the problem, 
nor ways to make the recommended "model" work better. He asks for blue- 
print details while we are still In the first stage of architectural plan- 
ning. If he chooses to call the recommended "model" an "idea" instead of 
a model, I don't quarrel with this. I did not set out to solve all prob- 
lems and to supply all details of problem solution. 

Nor can I fall to agree that It is liard to get conseneus in a 
pluralistic society. I only suggest trying to. No pol'tlcal party and 
no national leader has solved this problem. 

But I do predict that in the near future, the Office of Hducation 
will receive several proposals based on the consortium approach. It is 
for the authors of such proposals to spell out the mechanisms. 1 have 
no obligation to do that in the present paper. 

The critic complsL.s of too loose a depiction of the relation of 
development to diffusion, and he even objects to references made to 
sources of more detail on the development side. He says there are 
her ter sources on diffusion, but he does not name them. 

As a his^orlc4l fact, my paper was commissioned after the project 
leaders decided that the first drafts of the other papers did not repre- 
sent adequate coverage of development and diffusion. So with a late 
start, and no written feedback, I wrote two drafts of my paper, subse- 
quent to which the project staff put the two drafts together in the pre- 
.sent form, I appreciate this contribution by the project staff, and I 
absolve them from any responsibility for the present content of the 
paper, a-^ they did not change the substance of my own two drafts. 

The weaknesses in organization of the paper, so prominent in the 
criticism, wi'>uld have been even more pronounced without the help of the 
project staff during a period of Illness and other scheduling problems. 
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So my major question to the critic and to the reader is: **Can you 

propose a mori' viable model for development and diffusion?'* 

Siiu'o no ro\-/rlte of the paper is intended, as the critic was aware, 



for the tutme to prove whether the model will ever be tried out, and 



altliouip I <ln [)lan to conduct a Delphi study to see how educational 
derisi^i makers rc’art to tlie feasibility of this model for the future 
a.s ronj)ar*od to two otlier models reviewed briefly in my paper, and in 



I iiavo already vipo I o^; i;>:ed for not cltinp, the work of more developers 
and diffusers. A root note made clear that I was relying upon the 
Havelock volume tor more complete credits to father authors. The critic 
is perfectly correct in assuming that my own experience is closer to 
development than t'"* diffusion, lie has also said the task should have 
been divided into :wo papers. Perhaps he has proved his point. 

I do not agree that systems models work less well in education than 
in '’training.” The development model referred to so often as to irri- 
tate the critic has been applied by 165 graduate students in all teach- 
ing areas of education, often meeting the familiar 90/90 criterion of 
ef f ec t iveness . 

Clearly the critic and I differ on our degree of dependence upon 
rational analysis vs. empirical facts. The facts quoted from the 
Markle report are significant in themselves, apart from the fact that 
education often has a more heterogeneous population of learners than 
do many industrial or military training courses. 

So as to the development aspect of the proposed model, all I can 
say is: '*It works." As to the diffusion component, time will tell. 

T. coking to the future, I expect that several successful models 
will appear. For a while, old and new models may be applied simultan- 
eously. I do not say there is only one way to do it, and I did not 
say so in tlie paper . 

I do expect that consortium agreements in the proposed model would 
be few and minimal (and very general) at first, followed by increasing 
detail of consensus with each iteration or cycling of effort. For ex- 
ample, the first agreement might b^ that instructional materials should 
be learner oriented, not teacher oriented. Modular construction might 
be agreed upon to enhance individualization of obj ec tives and standards 
for different learners. Gradually more detailed goals or objectives 
would be agreed upon. I agree with the critic’s estimate that this is 
the main function of some organizations in the consortium, while others 
take the lead in development , evaluation, teacher training, and other 
components . 



and slnrc* i n* proposed "model" is my suggestion foi the future, it is 




work . No further cleba te can sc f t Ic this question , 




tfip much thicker volume by Havelock and his associates . 





But If 
I assume It 
agreements , 
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SYNOPSIS 



The Schalock-Sell paper also focuses on the nature of educational 
RDD&E activities, but from a point of view that is considerably differ- 
ent from that reflected in the other papers in the volume, The Intent 
of Schalock and Sell is to put forth a conceptual framework for viewing 
educational RDD&E that can be used to study those activities empirically. 
As a consequence, even though the paper covers the same ground as the 
other three, it does so in a way and at a level of detail that gives It 
a distinctively different flavor. 

Schalock and Sell view RDD&E activities as "problem-solving tools" 
which have icentifiable outputs. Accordingly, RDD&E are defined formally 
in terms of their outputs, their linkages are tracked in terms of outputs, 
work requirements are identified in terms of outputs, etc. Seven con- 
ceptual sets are provided in the paper, all of which have been developed 
for and applied to the description of ongoing educational RDD&E projects. 
The emphasis upon the output side of RDD&E is consistent with the point 
of view exprc;ssed by Gideonse, and seemingly consistent with the view of 
brlggs, though he doesn't speak of outputs as such. 
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Introduction 



"What's the use of their having names," the 
Gnat said, "if they won't answer to them?" 

"No use to them," said Alice; "but it's useful 
to the people that name them, 1 suppose. If 
not, why do things have names at all?"^ 



The purpose of the present paper is to present a set of understand- 
able words that describe the nature of research, development, diffusion, 
and evaluation activities as they occur within the context of education.^ 
Collectively the words are intended to represent an intact conceptual 
system or framework, and thus are presented as a carefully defined 
set of concepts and an equally carefully defined network of relation- 
ships between concepts. The intent of the framework is threefold: 

(£) to provide a way of looking at the processes of RDD&E so as to 
optimize their applicability to education; (^) to provide a structural 
basis for the empirical investigation of such processes as they are 
applied within the context of education; and ic) to provide a language 
for communicating about both (a) and (b) . 

Five propositions provide the basis for the paper and the conceptual 
framework that it outlines: 

1. The processes of research, development, evaluation, and 
diffusion represent an interdependent set of problem 
solving tools. 



^ John Kemeney prefaced the chapters of his book A Philosopher Looks 
at Science (Van Nostrand, 1959) with selected excerpts from Lewis CarroTlTs 
Alice in Wonderland . Various of the excerpts, such as the one above, fit 
the intent and circumstance of the present paper so well that they have 
been incorporated within it. Our thanks to Miessrs. Carroll and Kemeney. 

^ The formal definitions of these and other critical terms are in 
the GLOSSARY that will be found at the end of the paper. The meanings 
generally associated with them are introduced in the context of their 
discussion. When these terms are used as a collective set within the 
paper they are often referred to as RDD&E* 



2. These tools have proved useful in supporting change in a 
wide range of raan*s endeavors, for example science and 
industry, but they have not as yet proved to be particularly 
useful in support of change in education. 

3. Che tools of RDD&E hold considerable promise for the 
solution of probieias within education, but before that 
promise can be realized they must be better understood, 
more fully developed, and better adapted to the partic- 
ular requirements of education. 

4. The development and/or adaptation of RDD&E to meet the 
requirements of education can be accomplished best 
through empirical investigation. 

5. Prerequisite to empirical investigation is a conceptual 
structure that marks the boundaries of that which is to 
be investigated and identifies the elements within those 
boundaries that are to be subjected to investigation. 

As a point of departure in the paper, and as a way of making the assump- 
tions of its authors explicit at the outset, each of these propositions 
is discussed briefly. 

1. RDD&E As An Interdependent Set of Problem Solving Tools 

While RDD&E are not usually construed as tools, it seems plausible 
to think of them as such. According to Webster^s New World Dictionary , 
College Edition (1968, p. 1535), a tool may be thought of as "a means** 
to an end; and, since RDD&E are thought of as processes (activities, 
strategies) that produce outcomes, they meet the requirements of the 
Webster definition. So construed they are processes designed to ex- 
tend knowledge (the output of research), create technology (the output 
of development), generate information in support of decisions that 
bear upon the worth or effectiveness of a product, event, or condition 
(the output of evaluation), and lead to the adoption and/or utilization 
of the outputs of research, development, and evaluation by the various 
consumer groups that have need for them (the output of diffusion). 

As producers of outputs RDD&E serve education in many ways. In 
some instances their outputs may serve in and of themselves as solutions 
to problems faced by educators. In others they may serve as partial 
solutions to such problems, or as contributors toward solutions. The 
ability to link the outputs of RDD&E to the solution of educational 
problems seems to be largely a matter of how broadly or narrowly an 
educational problem is conceived, or how broadly or narrowly RDD&E 
outputs are defined. 

RDD&E are frequently seen as interdependent activities. The 
Icng standing dependency of development upon research, and its re- 
verse — the stimulation of research by the demands of development — 
are well known; the utilization of the outputs of research, develop- 
ment, and evaluation are obviously dependent upon the success of 
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diffusion activities; and, it is coming to be recognized (see, for 
example, Hemphill, 1969a and Schutz, 1970) that development, at 
least as it is now coming to be practiced within the context of ed- 
ucation, is as dependent upon evaluation as it is upon research* It is 
also coming to be recognized that both evaluation and research de- 
pend heavily upon development. Given these apparent interdependencies, 
and the assumptions that the extension of knowledge, the creation 
of technology, etc. are in fact solutions to problems, the view of 
RDD&E as an interdependent set of problem solving tools seems reason- 
able. 

2. The History of RDD&E in Education as Compared 
to Their History in Other Aspects of Man*a Endeavor 

One has only to point to the recent history of the aircraft and 
space industry, agriculture, medicine, and warfare, and trace the role 
of RDD6E within them, to gain some sense of the utility of these activities 
as problem solving tools in service of particular ends. A comparable 
history is difficult to trace in education. Educational development, 
evaluation, and diffusion, as recognized entities within a body of lit- 
erature, have existed for little more than a decade (Chase, 1970). And 
educational research, while reflecting a much longer history and a 
much larger body of literature, has been unimpressive in the difference 
it has made in the quality of education. In the view of Shulman (1970): 

"...If the object of such research is the development of co- 
herent and workable theories, researchers are nearly as far 
from that goal today as they are from controlling the weather. 

If the goal of educational research is significant improvement 
in the daily functioning of educational programs, I know of 
little evidence that researchers have made discernible strides 
in that direction" (p. 371). 

If these views are accurate, then in comparison with their contributions 
in other areas of man's endeavor, the net contribution of RDD&E to the 
improvement of educational practice has been small indeed. 

3. The Potential Contribution of RDD&E 
to the Improvement of Education 

While the overall contribution of RDD&E to education thus far ap- 
pears to be small, it Is the view of the authors that these processes 
represent potentially powerful tools for solving a wide range of prob- 
lems that face education, and that in time their potential in this 
regard will be realized. It is our belief, for example, that in time 
it will be possible to establish principles of instruction^ that can 
be used reliably in designing effective learning plans for children and 
that in time, through the application of systematic development proced- 
ures, it will be possible to create instructional systems that will 
bring about particular learning outcomes for particular students with 



3 

As used here a principle of instruction refers to a demonstrable 
relationship between sets or clusters of instructional strategies and 
the achievement of a particular learning outcome for a particular kind 
of learner in a particular kind of setting. 
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known degrees of reliability. It is also our belief that diffusion 
practices will be found that assure the optimal distribution, adoption 
and use of the outputs of research and development, and that evaluation 
procedures will be established that enable researchers, developers, dif- 
fusers, and practitioners to adapt that which they are attempting to do 
so as to make it optimally effective. The distance between hope and 
achievement, however, is great and it is our view that before such break- 
throughs occur the general problem solving strategies of RDD&E will have 
to be translated into tools that are more effective within the context 
of education than they are at present.^ 

A, Adapting RDD&E to the Needs of Education 
Through Their Empirical Investigation 

The proposition that the empirical investigation of RDD&E activities 
within the context of education provides the best possible means of 
bringing about their adaptation to the needs of education rests upon 
three related assumptions: (^) that to be effective the strategies of 

RDD&E have to be adapted to fit the particular demands of the particular 
context in which there is an application; (^) that by and large such 
adaptation has not as yet been made in education; and, (O that the 
best way to adapt any process to fit the demands of any context is 
to systematically study that process as it is applied within that context. 
Thus it is assumed that RDD&E within the context of the automobile indus- 
try take a somewhat different form than they do within the context of 
agriculture, and that to be effective they will take still a different 
form within the context of education. Moreover, it is assumed that as 
variation appears within any one of these broad areas of application — 
for example, the variation that exists between the professional school 
of a university, a traditionally organized high school, and an open 
space elementary school that emphasizes Individualized instruction — 
even further adaptations will need to occur. 

Such a view rests upon an appreciation of the power of contexts 
or situations to shape that which occurs within them (Brunswick, 1956; 
Shulman, 1970) and assumes that to be effective within specific con- 
texts the strategies, processes, and goals of RDD&E must be adapted 
accordingly. 

Assuming the correctness of such a point of view, how can RDD&E 
best be adapted to fit the needs of education? The proposition ad- 
vanced here is through turning the tools of RDD&E upon themselves. 

By systematically manipulating these activities within well defined 



It needs to be recognized that even when such translations have 
been made the resulting tools will not be robust enough to solve all of 
the problems faced by education. Projecting educational needs of the 
future, for example, or establishing sound educational policy or securing 
th i financial underpinings of education are problems that are not likely 
to yield to the tools of educational RDD&E alone. 



educational contextSi and systematically determining their effec- 
tiveness within themi it should be possible within a relatively short 
period of time for RDD&K to become the kind of adaptivei alternative- 
providing force in education that it has become elsewhere. 

5* The Dependency of Empirical Investigation 
Upon a Conceptual Framework 

Prerequisite to any empirical investigation is a set of concepts 
that marks the boundaries of that which is to be investigated and iden- 
tifies the elements within those boundaries that are to be the focus 
of the investigation.^ In their present stage of development educational 
RDD&E do not have conceptual structures that meet these criteria (support 
for this statement is provided in the next section of this paper) and, 
as a consequence, such structures have to be established before the empir- 
ical investigation that has been called for above can be undertaken. The 
primary aim of the present paper is to outline a first approximation to 
such a set of conceptual structures. 

So much for rationale. The paper consists of three parts: 
a review of the 11 terat ure pertaining to educational RDD&E, the presen- 
tation of the framework, and recomnendatlons as to steps to be taken 
to increase the effectiveness of RDD&E as problem solving tools in 
education. The bulk of the paper is devoted to the description of 
the conceptual framework. 

The approach taken to the development and presentation of the 
conceptual framework can best be described as at once analytic and 
provisional. It is analytic in that it starts with a set of generic 
definitions for RDD&E and on the basis of these definitions proceeds 
to lay out a conceptual frame that describes their nature and function 
within the context of education, the Interactions that occur between 
them, and the elements that comprise them. It is provisional in that 
while the framework has had continuous empirical referencing in the 
course of its development, it has not had the benefit of carefully 



5 As used in the present paper, concepts are logical constructions that 

'*. . . mark out the paths by which we may move more freely in 
logical space. They identify nodes or Junctions in the network 
of relationships, termini at which we can halt while pre-serving 
the maximum range of choice as to where to go next • • . There is 
a chemistry, as it were, for all things, and their scientific 
conceptualizations aim at identifying the elements and compounds 
of this chemistry . . . The function of scientific concepts is to 
mark the categories which will tell us more about our subject 
matter than any other categorical set" (Kaplan, 1964, p. 52). 

As such concepts represent the starting point in the extension of 
knowledge, and in many respects its ending point. 

^ See the third section of this paper (p. 212) for a review of the 
empirical testing that the proposed framework has received. 
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deoigned studies to test its validity. As a consequence the framework 
is offered only as an approximation to that which will emerge ultimately. 
Such a stance is taken purposefully, however, for from the authors' 
view of science all things aaaume a provisional status. In Kaplan's 
(1964) terms: 

"Every taxonomy is a provisional and explicit theory. 

. . As kn(V ledge of a particular subject-matter grows, 
our conception of that subject-matter changes; as our 
concepts become more fitting, we leam more and more. 

Like all existential dileionas in sc'ence, of which 
this is an instance, the paradox is dissolved by a 
process of approximation; the better our concepts, 
the better theory we can formulate with them, and 
in turn the better the concepts available for the 
next improved theory. . . It is only through such 
succession that the scientist can ultimately hope 
to achieve success" (pp. 53-54]. 



Notes on a Conceptual Muddle 



"It's really dreadful," she muttered to herself, 
"the way all the creatures irgue. It's enough 
to drive one crazy." 



In reading the literature that pertains to educational research, 
development, evaluation, and diffusion the overwhelming Impression 
gained Is one of conceptual disarray. The long standing distinction 
between basic and applied research Is under attack (Reagan, 1967; 
Carroll, 1969; Kidd, 1959); the similarities and differences between 
research and evaluation are still not clear (Cronbach and Suppes, 1969; 
Hemphill, 1969b; llayman, 1970; Schalock, 1970b); rival "models" 
of ivaluatlon are competing for attention (Stake, 1967; Stufflebeam, 
1968; Provus, 1969; Glass, 1971); the nature and purpose of educational 
development Is still under definition (Hemphill, 1969a; Jones, 1969; 
Schutz, 1970; Walker, 1970); and, the concept of diffusion Is still 
essentially without meaning. Sometimes It Is cost within the context 
of knowledge or research utilization (Jung and Llppltt, 1966; Havelock 
and Benne, 1967); sometimes within the context of the dissemination of 
information (Farr, 1969); and sometimes within the context of Innova- 
tion and change (Foshay, 1966; McClelland, 1968). There are also con- 
flicting views as to the nature of the interdependencies that exist 
between these activities. At times they are described In linear 
relationships (Cuba and Clark, 1965); at times in decision-oriented 
relationships (Cldeonse, 1968); and, at times In "linkage" rela- 
tionships (Mason and Boy an, 1968; Glaser, 1968). 

The purpose of this section of the paper Is to highlight some 
of the major conceptual problems that exist In the literature per- 
taining to RDD&E so that the reader may have a sense of the context 
within which the proposed framework rests. 

The literature reviewed in the following pages focuses essentially 
upon problems of definition. In this sense it is a very limited review 
and one that may have limited utility, since definitions are arbitrary 
and continuously under modification. For purposes of the present paper, 
however, such a review is seen as an appropriate place to begin, for 
it is by definition that meanings are specified and concepts born. Con- 
cepts emerge only after meanings are assigned to terms and the process 
of concept clarification, simplification, and systematization can 
occur only after an initial set of concepts exist. Relying upon Kaplan 
(1964) once more; 

". . .the process of specifying meaning is a part of the process 
of inquiry itself. In every context of inquiry we begin with 
terras that are undefined — not undef ineables, but terms for which 
that context does not provide u sped'’ nation. As we proceed. 




2»7 



203 



empirical findings are taken up into our conceptual structure by 
way nf new specifications of ineaning, and former indications and 
references in turn become matters of empirical fact" (p. 77). 

Thus, beginning with definitions is very much like beginning at 
the beginning. Aid, in a veiw of the literature on educational RDD&E, 
that seems like a good place to begin. 

Research 



Of the four basic concepts with which the paper deals, that of 
research seems to have the clearest meaning. By and large it i;; seen 
as a set of activities governed by a reasonably well specified set of 
procedures that lead to the extension of knowledge, with knowledge 
being defined broadly as conclusions that can be logically defended or 
empirically demonstrated. Beyond this general definition, however, 
there is little agreement to be found. There is still no clear agree- 
ment, for example, as to the relationship of research to the broader 
concept of "disciplined inquiry" (Cronbach and Suppes, 1969), or as 
indicated earlier, even to the long standing distinction between basic 
and applied research. *v^en the criteria that have been used to make 
such distinctions are viewed closely, however, this is not surprising.^ 
They imply that knowledge produced in an attempt to control or eradicate 
cancer, build an atomic bomb, reach the moon, or make it possible for 
every child to read (all instances of "applied" or "mission-oriented" 
research) is in some way fundamentally different from or is produced by 
fundamentally different means than knowledge produced in an attempt to 
understand the molecular structure of cells, the structure of the atom, 
the origin of the earth, or how children learn (all instances of "basic" 
research) . While it may be true that the motives of persons working 
toward the solution of such problems differ, and that the variables 
manipulated and the methodologies employed differ, the bothersome 
question is whether the nature of the magnitude of such differences is 
so great that separate label- have to be created to distinguish between 
them. 



^ See for example the definitions offered by Cohen (1948) and the 
National Science Foundation (1960), respectively. 

"The difference between those who work at fundamental research 
and those who work at applied research is in the point of view 
with which they face the problem and the goals they ha.e in mind. 
The man working at the *pure science* end of the spectrum, whether 
in a university or in an industrial laboratory, pursues a problem 
because it is interesting or because it appears to have a certain 
relevance to fundamental knowledge. By contrast, the man working 
at the applied science end of the spectrum pursues a problem be- 
cause it has relevance to a particular practical goal" (p. 303). 

"i Vasic research is directed toward increase of knowledge; it is 
rereaich where the primary aim of the investigator is a fuller 
’understanding of the subject under study rather than a practical 
application thereof. Applied research is directed toward prac- 
tical app.1 ; cations of knowledge. (Applied research projects) have 
specific commercial objectives with respect to either products or 
processes" (p. 5). 



In some respects questions as to the nature of differences between 
kinds of research appear to be of a ”how-many-angels-on-the-head-*of-a-pin" 
variety • What does it matter that some research is directly applicable 
to needed solutions to problems while some is not, or that some people 
are interested in cue extension of one kind of knowledge while others 
are interested in the extension of another? Isn't the extension of 
knowledge the extension of knowledge? And aren't the basic rules 
governing that activity roughly the same across substantive areas, or 
across problems within a particular area? 

In other respects, however, it may be that differences really do 
exist in the conduct of different kinds of research, or that different 
temperaments really are needed by people to carry out different kinds 
of research, o: that different strategies do have to be followed to 
secure funding for different kinds of research. If this should be the 
case, then questions as to the nature of the differences within that 
which can properly be labeled research may serve a useful function. 

And that seems to be where the definitional problem resides. On 
the one hand, research is research; on the other, there is the gnawing 
sense that there are fundamental distinctions between kinds of research. 
The basic and applied distinction has been around for a long while, and 
funding agenc:(es, state legislatures, and the Congress act in terms 
of it, even though they are not particularly comfortable with it. But 
efforts to bring other distinctions forward — for example, mission 
oriented or operations or action research — have not seemed to solve the 
problem. 

In the proposed framework the general definition of research as a 
set cf activities that leads to the extension of knowledge has been 
accepted. Within this general definition, however, attention is 
directed to the question of whether there are not other dimensions on 
which differences in research activities can be ordered that will 
give them greater functional utility within the context of education 
than the basic-applied distinction. 

Evaluation 



While the term evaluation is as familiar to persons in education 
as is the term research^ it by no means has as much agreement as to 
definition. The first and perhaps the most basic source of disagreement 
is whether evaluation is to be thought of as an "information” generating 
activity or a "knowledge” generating activity. The models of evaluation 
proposed by Stake (1967), Stufflebeam (1968; 1969), and Provus (1969) 
assume that the purpose of evaluation is to collect information that 
is to s'^rve on-line decision-making, without regard to the generalizability 
of that information. The models proposed by Suchman (1967) and 
Cronbach and Suppes (1969), however, argue that procedures should be 
employed in the collection of such information so as to make it general- 
Izable. 

In the view of the authors, it is this difference of opinion that 
leads to much of the confusion that exists in the literature on eval- 
uation (Cronbach and Suppes, 1969; Hemphill, 1969b; Hayman, 1970; 
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Schalock, 1970b), and to much of the language that reflects such 
confusion. The recent article by Caro (1971) illustrates the problem 
(italics have been added): 

"Efforts in the past decade to reduce the incidents of major 
social problems have stimulated a new interest in program 
evaluation . Numerous writings have appeared on the use of the 
concepts and methods of behavioral research in evaluating these 
interventions. This paper is a review of that literature. . . 

The first part of the paper is concerned with the basic issues 
which include definiLions, approaches to evaluation me thodology , 
roles of evaluation in program development , and distinctions 
among various forms of research . The second section deals with 
organizational matters such as the establishment of the eval uative 
research role , administration of evaluation research , utilization 
of the results of evaluation , and implication of client activism 
for evaluation. . ."(p. 87). 

Sentences appearing later in this article, such as, "Some of the non- 
use of evaluation results is attributable to limitations of the research 
itself " (p. 95), do not help the matter. 

While it is not possible to tell whether the apparently confused 
but surely confusing language in the Caro article is a matter of fuzzy 
concepts or an imprecise use of terms, the effect on the reader is 
the same. 

Attendant to the issue of whether evaluation is a knowledge 
generating activity or an information generating activity in service 
of on-line decision-making is the issue of similarities and differences 
in methodology. Suchman (1967) argues that the distinction between 
basic research and what he calls evaluative research is one of purpose 
rather than of method. Hemphill (1969b) holds a similar view. Cuba 
and Stufflebeam (1968) argue, however, that this is not so, claiming 
that the application of comparative experimental designs to evaluation 
problems makes unwieldy the collection of data, and thus reduces the 
possibility of the evaluation function to effectively support on-line 
decision-making. The entry of others into the debate (Glass, 1971; 
Stanley, 1969; Hayman, 1970; Schalock, 1970b) mark the issue as one 
which will not be easily or quickly put to rest. 

A host of other distinctions and issues contribute to the defin- 
itional problems surrounding the concept. As in the case of research, 
evaluation has its own version of the dichotomy dilemma in its distinction 
between formative and summative evaluation . (Scriven, 1967).® It 
also has competing "models" of evaluation, for example, those proposed 
by Stake (1967), Suchman (1967), Stufflebeam (196S), and Provus (1969), 
each of which seems to derive from somewhat different views of the 
nature of evaluation. There are also distinctions which emerge as 



o 

As used by Scriven formative evaluation is designed to improve 
a product or program while it is still fluid; summative evaluation is 
designed to appraise a product or program after it is well established. 
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a consequence of the ''focus” of a particular evaluation. Levien 
(1970) identifies four distinctively different foci for evaluation ac- 
tivities — policy evaluation, program evaluation, outcome evaluation, 
and assessme,nt evaluation — each of which apparently requires somewhat 
different methodologies and each of which overlaps or crosscuts all 
of the other distinctions made above. 

Finally, there is disagreement as to whether the act of valuing 
or judgment-making in the process of evaluation should be done by 
the "evaluators” or left to the "decision-makers." The preference 
of Scriven (1967) and Glass (1971) is for the evaluators to do the 
evaluating. The models proposed by Stake, Stufflebeam, and Provus 
prefer that the valuing activity be carried out by those who are to 
use the information generated by the evaluators. 

Given such wide ranging differences in point of view as to the 
nature, function, and focus of evaluation it is hard to know where to 
begin with definitions. On the assumption, however, that as yet 
evaluation has no clear cut identity apart from research the major 
concern within the proposed conceptual framework is in making that 
distinction. Attention is also directed to the relationship of eval- 
uation to development and diffusion, and to the matter of differing 
kinds of evaluation; but, the framework does not carry the latter 
set of distinctions far. 

Development 

Richard Schutz, a leading practitioner of that which is called 
educational development, has recently written that: 

"Writing on the nature of educational development at the present 
time is as speculative as writing on the nature of the industrial 
development would have been in the 17th century. The scope of 
the phenomenon, 'educational development,* has yet to be realized" 
[1970, p. 39]. 

While Schutz *s comment is probably as true for the methodology of educa- 
tional development as it is for its scope, there is a body of literature 
which suggests that there is at least a beginning awareness of what the 
field of educational development might be and the form which its method- 
ology might take. In reading its literature, however, one gets the 
impression that the field is not as far in its evolution as that of 
evaluation and, as a consequence, the conceptual base within it is 
even more fragmentary. 

In general terms educational development is seen as "a systematic 
process of creating new alternatives that contribute to the improvement 
of educational practice" (Hemphill, 1969a, p. 23). This definition is 
consistent with the views of Hamreus (1969), Jones (1969), Randall 
(1970), and Schutz (1970). It is also consistent with the definition 
of development provided by the National Science Foundation: "The 

systematic use of scientific knowledge directed toward the production 






207 



of useful materials, devices, systems of methods. . (1969, P. 2). 

The emphasis upon systems design principles of nearly ell current 
writing on the subject has led to the methodology of educational 
development being described variously as knowledge-based, systematic, 
rigorous, empirical, data dependent. 

It is this dependency upon the use of data to determine the ef- 
fectiveness of that which is being developed which sets systematic 
educational development .apart from the developmental efforts of 
teachers in the schools , publishers , national curriculum groups , 
and the like. As Jones (1969) has put it: 

"It is at once apparent that numerous groups such as publishers, 
national curriculum groups, and professional curriculum groups 
have endeavored to develop more appropriate rationale and mater- 
ials for learning. Yet, absent from most of these endeavors 
is one essential developmental ingredient — testing the products 
to determine if they actually achieve the objective for which 
they were developed" 1) . 

Beyond these broad lines of agreement about the nature and function 
of educational development, there is little that seems to match in 
any precise way. The focus of development varies from hardware to 
software, from product to process, from materials that support the 
learning of children to materials that support the learning of teachers 
of children. The specific steps to be followed in the various models 
of development seem to vary with each group that is involved in ^he 
developmental process (Briggs, 1972). And the language within which 
the data-dependent mode of operation is couched shares little agreement. 

As in the case of evaluation, the primary concern with the con- 
cept of development in the proposed conceptual framework is its 
distinction from the concept of research. Attention is also directed 
to the interdependencies of development, evaluation, and diffusion and 
to differences in kinds of development. But as in the case of eval- 
uation the framework does not carry the distinctions far. 

Diffusion 



Of the four basic concepts with which this paper deals, the concept 
of diffusion is the least clear in its meaning in the literature. As 
indicated previously, sometimes the concept is couched in terms of 
dissemination, sometimes in terms of planning for innovation, and 
sometimes in terms of 'viowlrdge or research utilization. In some cases 
writers have proposed diffusion as the process that is designed to get 
the outputs of research, development, and evaluation into the hands 
of users, by whatever means (Havelock and Benne^ 1967), while others 
prefer to identify discrete steps within that overall process and to 
refer only to some of those steps as inclusive of the domain of dif- 
fusion. Cuba and Clark (1965), for example, divide the activities 
that lead to the utilization of research, development, and evalu- 
ation outputs into two major headings — diffusion and adoption. 

Ur.der diffusion they include the processes of dissemination and 
demons t rat ion , and under adop t ion they inc lude the pro cesses of 
trial, installation, and institutionalization. 



other writers hold differing views. For example, Hemphill’s 
(1969a, p. 24) notion that the product development process tends 
to equate dissemination with the functions of manufacturing and 
marketing, and Briggs (1972, p. 149) notion that developed products 
serve as the change-creating mechanisms. 

Given such a state of affairs, the conclusion reached by Havelock 
and Benne (1967) after an extensive review of the literature in the 
area of research of knowledge utilization is not surprising: 

"While the utilization of knowledge is a hope for some and 
even a firm expectation for others, it remains, for most 
of us, a poorly defined and poorly articulated concept. On 
the one hand we are aware of an enormous and ever-increasing 
body of specialized scientific knowledge and, on the other 
hand, we have a vague vision of this knowledge being used 
to make better w^^shing machines, safer automobiles, better 
schools, and happier community relations. Yet there is no 
clear picture of how we get from one end of the utilization 
chain to the other" (p. 49). 

Nor is it surprising that Hood (1972) reaches a similar conclusion 
with respect to the diffusion of hard products. 

In the authors' judgment major sources of confusion in the 
literature on diffusion rests with the failure of writers to dis- 
tinguish clearly between kinds of research, development, and evalu- 
ation outputs to be diffused, for seemingly the nature of that 
which is to be diffused influences significantly the processes by 
which diffusion occurs. When that which is to be diffused is the 
output of rejearch or evaluation activities, i.e., knowledge or 
information, it seems appropriate to talk of "dissemination" as 
the appropriate vehicle to support diffusion efforts. In this case 
dissemination refers essentially to the distribution of information 
to potential consumer groups. 

If that which is to be diffused is an output of development, 
however, such as a new curriculum, a new assessment device, or a 
new scheme for scheduling students into classes, the process of 
diffusion apparently requires steps beyond a simple distribution 
of information.^ In diffusing outputs such as these there must 
still be the dissemination of information about them, but apparently 
there must also be (a) an opportunity to see the product in operation 
(a demonstration), (lb) an opportunity to try it out, (O an opportunity 
to weigh information relative to cost and resource requirements for 
its implementation and long-term operation, (^) support services 
to help in its initial installation and operation, and (^) the 



^ Cuba (1967), in keeping with distinctions made earlier, (Cuba 
and Clark, 1965), insists that diffusion and adoption be treated as 
separate entities. For purposes of the present paper, adoption is 
treated as one of the terminal steps in the diffusion process. 



proviaion of support services for long-term maintenance and operation. 

It La aiao probable that diffusion strategies need to vary according 
to the specific features of a product under consideration. For example, 
the larger the change that the adoption of a given product will make 
upon an operating setting, the greater the care that is likely to be 
needed in relation to all of the considerations just outlined. 

In the conceptual framework proposed in the present paper attention 
is directed to the nature of the diffusion process, and the relationship 
of that process to research, development, and evaluation. Some attention 
is also directed to differences in strategies of diffusion, depending 
upon that which is to be diffused. As indicated previously, the rationale 
for treating diffusion within the same framework that research, develop- 
ment, and evaluation are tireated is that diffusion i*^ the means by which 
the outputs of research, development, and evaluation are to find their way 
into practice. Another reason for doing so, however , is the author's 
belief that Hood's analysis is correct: "Educational diffusion may 

become the Achilles' heel of the entire research, development, and 
evaluation enterprise" (1972. p. 130). 

And Various Combinations Thereof 



In the Introduction to the paper the processes of RDD&E were referred 
to as an interdependent set of problem solving activities or tools. Within 
that context the obvious linkage of diffusion to RD&E and the much dis- 
cussed relationship between research and development were mentioned. 

Also mentioned was the growing awareness of the dependency of develop- 
ment and diffusion upon evaluation, and the dependency of evaluation 
and diffusion upon development. The point of that discussion was to 
establish the validity of viewing RDD&E as an interdependent set of 
problem solving activities. The purpose of the present discussion 
is to provide a brief review of the various ways in which these inter- 
dependencies have been viewed by persons writing in the field. 

Gideonse (1968; 1969; 1970; 1972), probably more than any other 
writer, has worried about the nature of the interdependencies between 
RDD&E and the way that they are conceptualized. In the report of a 
major review of educational research and development in the U.S. (1969), 
he identifies three "models" that describe these interdependencies. One 
of these he labels as a linear or dependency model which ". . .tends to 
view the goal of educational improvement as being dependent upon ad- 
equate diffusion mechanisms, which in turn require the invention and 
development of tested innovations to diffuse, which in turn depend upon 
the adequacy of the research base" (p. 5). Cuba and Clark (1965) are 
cited as persons who support a linear model of RDD&E.^® 

The second and third "models" of RDD&E in education identified by 
Gideonse (1969) are the decision-oriented and linkage models. He sees 
fhe decision-oriented model as holding 



It should be noted, however, that in their 1965 paper, and 
subsequent (Clark, 1972), these authors have insisted that the linear 
model that they have proposed is only an idealization or abstraction. 
They do not assume thcX RDD&E necessarily operates in such a manner. 




• .essential differences and disconnections between the research, 
development, and dissemination functions. Models such as these 
draw attention to the different rules of evidence and sources and 
types of data Input to decision >naking in each function. The 
relationships among different types of activities within research 
and development are recognized, but these models tend to be 
more impressed by the present day decision-making requirements 
than by patterns which may emerge from somewhat longer-term 
historical analysis of change or from the apparent logical 
dependence of one function on another" (p. 5). 

and the linkage model as stressing 

"The close interrelatiions of research , development , and dissemina- 
tion. The linkages are elevated to closer scrutiny without nec- 
essarily limiting attention to the particular stages in inquiry, 
development, and dissemination. Models in this category may have 
a tendency to be performer-oriented and to stress the importance 
of individuals in a research-development-dissemination continuum" 

(p. 6). 

The work of the House of Representatives Committee on Government Opera- 
tions (1966) and Sherwin and Isenson (1967) are cited as examples 
basp.d upon or implying the decision-oriented model. The publications 
of Mason and Boyan (1968), Glaser (1968), and Raisbeck (1965) are cited 
as representives of a linkage model. 

In the authors' view there is validity In all three of the models 
identified by Gideonse. However, the framework proposed in the present 
paper reflects more closely the characteristics of the decision-oriented 
and linkage models than it does the linear model* This stems largely 
from the view that there is little that is predictable in the patterning 
of RDD&E when they are viewed as an interdependent set of problem solving 
activities. In our observations, the nature of the problem being pursued, 
the preferences of the persons involved, and the resources available to 
them seem to be much more powerful determiners of the sequence or pattern 
of activities which characterize a given problem solution than is a pat- 
tern which people assume to be implicit in such activities* 

The focus of the preceding has been a review of the state of concepts 
in the field of educational RDD&E* The framework which follows is intend- 
ed as a constructive response to it. 
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A First Approximation to an Empirically 
Based Framework for Viewing Educational RDD&E 



any of them can explain it,” said Alice, ”1*11 
give him sixpence, I don’t believe there is an atom 
of meaning in it.’^ 

"If there is no meaning in it,” said the King, "that 
saves a world of trouble, you know, as we needn’t 
try to find any," 



The framework that is outlined in the pages which follow represents 
an effort to provide a standard set of meanings for the concepts of 
educational RDD&E and their interactions. The framework consists of 
formal definitions, relational terms, and a taxonomic structure that 
provides a language for describing both the vehicles or mechanisms 
that carry RDD&E activities (projects, programs, institutions) and 
the structure, function, and operation of those mechanisms. Specifi- 
cally, the framework consists of seven interrelated category sets: 
one which provides basic definitions for educational RDD&E; one which 
provides for the description of the interactions between RDD&E activ- 
ities as they occur within the context of education; one which provides 
for the description of vehicles which carry educational RDD&E; and, 
four which describe the organization and operation of such vehicles. In 
combination these seven category sets are intended to identify all 
of the major dimensions of, elements within, and interactions between 
RDD&E activities as they occur within the context of education. 

Assuming that such exhaustiveness has been realized and that 
other critical properties of category sets have been obtained — for 
example, meaningfulness, power, and discreteness — the framework as 
a whole should provide the conceptual tools needed to describe or map 
the domain of educational RDD&E, or any given R,D,D, or E activity 
that takes place within it.^^ It also provides a language that permits 
reliable communication about that which is mapped and a way of thinking 
about RDD&E generally that should facilitate their more effective ap- 
plication within the context of education. 12 



11 It is probable that the category sets that comprise the framework 
are applicable to the description of RDD&E activities in fields other than 
education, but as yet they have not been tested in that regard. 

12 

The reader needs to be clear that the intent of the present paper 
is to present only the conceptual tools for mapping the domain of educa- 
tional RDD&E, not an actual map of that domain. Volume I in the series of 
volumes reporting the Oregon Studies contains such a map. Volume V in 
ihat series contains the operational definitions, decision rules, etc. 
that permit the conceptual tools described in the present paper to be 
fip')Iied in the mapping process. The volumes that comprise the series as 
u whole are described on the inside of the back cover. 



Finally, the framework as a whole provides a basis for a science 
or technology or discipline of educational RDD&E, in that it is addres- 
sed to; (ai) an inventory of the entities, phenomena, and relations in 
the field of inquiry, (jb) naming and describing those entities, phenomena, 
and relations, and (O classifying them in orderly systems (Dussault, 

1970, p. 12). According to Northrop (1947) and Nagel (1961) these are 
the conditions necessary for any scientific undertaking, for they are 
prerequisite to either the formulation of comprehensive theoretical 
systems cr the empirical investigation of the elements that comprise 
such systems. In the language of Kaplan (1964) they also represent 
the beginning of a theory in that they provide ". . .a way of making 
sense of a disturbing situation so as to allow us more effectively 
to bring to bear our repertoire of habits, and even more important, 
to modify habits or discard them altogether, replacing them by new 
ones as the situation demands" (p. 295). 

The stimulus to the development of the framework was the project 
from which the present volume em^irged (Schalock, 1970c) . As initially 
conceived the Oiegon project was desir ed as an "applied" research 
study that would directly support the efforts of the Research Training 
Branch of the National Center for Educational Research and Development, 
U.S. Office of Education, to design and develop new training programs 
for the preparation of educational RDD&E personnel. As the study was 
implemented, however, it assumed the properties of a basic research 
project in that its primary charge became one of providing (^) an 
initial mapping of the domain of e ".ucational RDD&E, (^) a methodology 
that would enable such mapping to occur, and (O a conceptual framework 
that would give meaning to that which was mapped. 

Specifically, the charge of the project became one of developing 
a methodology which would permit the detailed description of ongoing 
research, development, diffusion, and evaluation projects and/or pro- 
grams in education, and to apply that methodology to an illustrative 
set of approximately 20 such projects that cross-cut the range of 
ongoing educational RDD&E activities in the nation. The decision to 
select projects that ranged widely across the domain of educational 
RDD&E rested upon the need to identify the parameters of such activ- 
ities, and to insure that the methodology that was being developed 
would in fact be sufficiently generalizable so as to be applicable 
to the description of the full range of activities found within 
that domain. 14 



13 Hereafter referred to only as projects. 

14 A special tribute needs to be paid to the far-sightedness and 
courage of the advisory committee to the project in this regard, for 
they had the wisdom to encourage the evolution of the methodology 
throughout the life of the project. By so doing they placed greater 
faith in the long-term good that would result from a strong methodology 
than in the short-term good that would derive from the availability of 
data from a weak methodology. This meant. In effect, that the data from 
the 20 case studies would not be fully comparable. The trade-off was a 
more useful methodology for a more limited initial mapping of the field, 
a trade-off in the writers* judgment that was in favor of the long-term 
good of education. 
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The framework that is outlined in the pages which follow, then, 
is the result of a year’s effort at conceptualizing the nature of 
educational RDD&E aud testing that conceptualization against opera- 
ting projects. The testing engaged in, however, was only develop- 
mental or "formative” in nature, that is, versions of the framework 
were fashioned and then tested to see whether they described ade- 
quately that which occurred in the field, refashioned on the basis 
of that evidence, and recycled. In the course of the year, six major 
revision cycles took place. The framework has not as yet received 
formal empirical testing, and thus it is almost certain that it will 
need further revision as it is honed against formal hypotheses or 
as it is applied to ongoing projects that vary in some significant 
way from those on which it has been tested thus far. 

Conceptual Set 1. Definitions 

In the development of a framework that is intended to invite and 
support empirical investigation, care must be taken to insure that 
basic definitions not only clarify but perToit reasonable operational 
definitions to follow. As a means to this end, research, development, 
diffusion, and evaluation have been defined throughout the frame- 
work in terms of the outputs that they are Intended to produce. OUT- 
PUTS refer to identifiable and significant j)lanned outcomes of targeted 
work activities, with targeted work activities being actions directed 
toward the realization of projected goal states > 

Such an approach to the problem of definition is intended not only 
to sharpen the distinctions between RDD&E activities, but also to 
highlight the manner in which such activities — either individually 
or collectively— can contribute to the improvement of education. 

By fixing attention to the output side of RDD&E efforts the users 
of such tools should be able to be clearer about what problems they 
can hope to solve with them and what they can expect from their 
application. It is also assumed that by gaining greater clarity 
as to the outputs to be expected from the application of RDD&E in 
education, those who perform RDD&E services to education will be 
held increasingly accountable for producing those outputs. 

In the present paper first level or "generic" definitions 
of research, development, diffusion, and evaluation are offered. 

The concept of second level or "subclass" definitions is introduced 
and illustrated, but since the project from which the framework emerged 
did not require a second level analysis of RDD&E a corresponding 
conceptual frame has not been developed or tested. The direction of 
energy toward this end, however, is an undertaking that should assume 
priority in the years ahead. 

First Level Definitions 



Educational RDD&E . A coordinated set of problem solving strategies 
desi g ned to produce recognizable outputs that can be judged as to their 
^ [uallty and their contribution to the solution of educational problems , 
•^o defined, educational RDD&E has two major characteristics: (a) it 
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tends to focus upon the reduction of discrepancies — theoret J cal , method- 
ological, or operational in nature — between present and goal conditions 
in education; and, (^) it has as its consequences outputs that can be 
judged as to the adequacy with which they reduce the problem to which 
they have been directed, that is, the adequacy with which they reduce 
the discrepancy between present and goal conditions. The outputs ex- 
pected to derive from educational RDD&E are specified in the definitions 
which follow. 

Research , A problem solvlag strategy designed to produce reliable 
knowledge, that is, facts, principles, generalizations, theories, and 
laws that are generalizable and that can stand. the test of empirical 
verification , 1^ So defined , educational research is characterized by 
procedures of problem stating, design, measurement, and analysis that 
when applied produce outputs which have known degrees of generaliz- 
ability, and that can be replicated for purposes of verification. 

Criteria for determining the quality of an output that derives from 
a research effort involve a combination of judgments as to the 
extent to which the output reduces the discrepancy between present and 
goal states, and the extent to which confidence can be placed in it as a 
consequence of the steps taken in its derivation. Examples of outputs 
that derive from educational research include: evidence as to the ef- 

fectiveness of alternative instructional conditions in bringing about 
classes of outcomes for given sets of learners; the effect on teacher 
and student morale of a transition on the part of a school to team 
teaching; the effect of institutionalizing individualized or person- 
alized instructional programs in an elementary school on teacher-teacher, 
teacher-administrator, teacher-parent, and teacher-pupil interaction. 

Evaluation . A problem solving strategy that produces trustworthy 
information to be used in situation-specific decision making, that is , 
observations, reports, and other outputs of formal or informal measures 
that are collected with attention to the control of unwanted variance 
and organized so as to optimize their utility for a particular set of 
decision makers who have a particular decision or set of decisions to 
make within a given time frame . 1^ So defined, educational evaluation 



This is a relatively narrow definition of research if the inquiry 
related activities of the mathematician, historian, and philosopher are 
also to be considered. For a discussion of disciplined inquiry broadly 
conceived see Cronbach and Suppes, 1969. 

This definition avoids the issue of where the '‘valuing" of the 
information is to occur, at the point of the decision or at the point 
of preparing the information for the decision-maker. It does make clear, 
however, the distinction between research and evaluation. Research has 
an output (knowledge) that is generalizable; evaluation has an output 
(information) that looks very much like the output of research, but 
differs in that it is collected for purposes of situation-specific 
decision-making. As a consequence, most Information generated through 
evaluation activities is not generalizable. It is not, however, the 
result of "sloppy" research. 



o 






215 



1h characterized by a eet of procedures which generate in format Ion ~- 
through formal or Informal measures — that Is collected with as mucli 
attention to the control of unwanted variance as the constraints of 
time and resources will allow and that is organized for optimal 
use in decision-making. Judgments as to the quality of the outputs 
that derive from an evaluation effort center around the utility of 
such outputs, that is, their appropriateness to the decision to be 
made, their trustworthiness, and their timeliness. They do not have 
to meet the criteria of reproducibility or empirical verification 
that accompany the outputs of research, and they do not assume the 
property of generalizability. Examples of outputs that derive from 
educational evaluation Include: means, standard deviations, and 

Individual pupil profile gain scores in selected subject areas In the 
5th grade of school A; coded interview data from teachers and pupils as 
to the feelings that they have about the adoption of an IPl mathematics 
program In grades 1 through 3 in school B; teacher ratings of the 
extent to which a principal or a group of principals seek and Incorporate 
their opinions In policy decisions affecting the operation of a school; 
ncnobtruslve measures of the efficiency of the purchasing office 
of a school district, such as number of requisitions processed per 
day or the average length of time elapsed between submission of a 
requisition to the business office. Its processing, and Item delivery 
to a school. 

Development . A problem solving strategy designed to produce reliable 
technology, that Is, procedures, materials, hardware, and organizational 
f rameworks that have a known degree of success in bringing about a part - 
ticular outcome or in performing a defined operation .*” So defined, ed- 
catlonal development is characterized by a set of procedures which system- 
atically take outputs that are to be developed from their initial conception 
and design through a series of fabrications and operational field trials 
until there is empirical evidence that the output does that which It Is 
supposed to do. Judgments as to the quality of the output arc dependent 
upon evidence presented in support of its effectiveness in realizing the 
outcome, expected from it, the results obtained by others who have used it, 
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This is not to Imply, however, that the outputs of educational 
evaluation are without value to persons beyond the specific decision- 
making situation for which they are collected. They have value Just as 
case history Information has value to the practicing clinician, or as 
the experiences of a practicing I'esearcher have value to the student of 
research. They do not have the same kind of value, however, that comes 
from the power of data that generalizes within parameters, across sltua- 
Lons . 

18 

Note the similarity between this definition of technology and 
that printed In a recent report of the Brookings Institute (Nelson, et al. , 
1967): "Technological knowledge is a set of techniques, each defined 

as a set of actions and decision rules guiding their sequential appll- 
r .itlon, that man has learned will generally lead to a predictable (and 
sometimes desirable) outcome under certain specified circumstances. 






and the adequacy of the procedures used in obtaining the evidence that is 
reported for it. Examples of outputs that derive from educational de- 
V(!lopment include the IP I mathematics program » the instructional programs 
emerging from Project PLAN, and the Westinghouse Learning Corporation’s 
computerized program for the processing of test data. 

Diffusion . A problem solving^ strategy designed to bring about the 
adoption and/or utilization of generalizable knowledge, reliable tech -* 
nology , o r trustv o rchy information . So defined, educational diffusion 
is characterized by a set of procedures which provide information about 
and create the necessary conditions for the adoption and/or utilization 
hy appropriate consumer groups of outputs that derive from educational 
rcsearcli, development, and evaluation. Dissemination of information about 
a given output can be done through oral, written, pictorial, or electronic 
means, or any combination thereof. Activities leading to adoption can 
involve illustration, demonstration, adaptation to fit the requirements 
of an adopting context, installation on a trial base, or any combination 
thereof. Criteria for Judging the quality of an output that derives 
from a diffusion effort, that is, criteria for Judging the extent or 
success of adoption and/or utilization, involve a combination of 
Judgments as to the extent to whicti target populations have made adoptions 
and the success of operation once that which has been adopted has been 
installed. 

Second Level Definitions 



Providing definitions of educational research, development, evaluation, 
and diffusion that make it possible to identify and label such activities, 
and differentiate outputs among them, is a necessary place to begin in 
mapping the domain of educational RDD&E. If left at that level of general- 
ity, however, the map that would be produced would have limited utility. 

When attempting seric is study of activities as complex as those being dealt 
with, when attempting to carry out such activities, or when attempting to 
prepare others to carry out such activities, the limitations of generic 
definitions become readily apparent. Finer discriminations, if at all 
possible, should be made. Xu thp paragraphs ^hat follow some tentative 
notions are advanced as to the form which such discriminacions might take. 

Historically, with the exception of the long-standing distinction 
between basic and applied research, and the rather recent distinction 
between formative and summative evaluation, there have not been distinc- 
tions made between desses of activity within research or development 
or evaluation or diffusion. T!ie position taken here is that such 
distinctions should be made and that the organization and operation of 
educational RDD&E activities should be studied as to their variation 
across wlthin-class distinctions. While unMkely, it is possible that 
would find as much variation in the operations required to implement 
different classes of researen or development or evaluation or diffusion 
activities as one would find in implementing research as contrasted to 
development, evaluation, etc. 

In Older to undertake comparative studies of this nature, at least 
binary classifications within RDD&E are needed. Tlie long-standing 
dichotomy of basic and applied research, and the more recent distinction 
of formative and summative evaluation of course constitute binary 
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distinctions; but, what is being suggested here is the need for a 
redefinition of the binary sets. (The second section of the paper, 
"Notes on a Conceptual Muddle, " presented the rationale for this po- 
sition.) The definitions appearing in Table 1 represent an illustra- 
tion of such a set for resea; '"h. Definitions provided later (Tables 
3, 4, and 5) provide possible sets fot evaluation, development, and 

dif fwSion, respectively .19 



TABLE 1 

Illustrative Second Order Definitions 
for Educational Research 



Construct Ascendant Research 



Treatment Ascendant Research 



Research which focuses upon the 
identification or clarification 
of constructs, their parameters, 
or their interrelationships 



Research which focuses upon the 
identification or clarification 
of treatment conditions, their 
parameters, or their effects 
upon targeted outcomes 



Examples: Projects which attempt 

to identify the dimensions of 
’’creativity; the nature of 
’’observer influence;” the dis- 
tribution of ’’intelligence” 
within a given population. 



Examples: Projects which attempt 

to identify the influence of 
early separation experiences on 
interpersonal trust, or the 
effectiveness of alternative 
instructional strategies for a 
given class of children in relation 
to given class outcomes. 



Four observations about Table 1. First, the notions of construct 
ascendant and treatment ascendant research represent a substantially 
different way of looking at classes of activity within research than 
does the traditional basic-applied distinction. Also, they are in 
keeping with the general strategy of defining RDD&E in terms of their 
outputs. In the case of construct ascendant research, the output Is 
the clarification ur identification of the parameters of conceptual 



The reader needs to be aware that the second level definitions 
offered in Tables 1, 3, 4, and 5 are intended ^nly as illustrative, 
and that they have not had careful conceptual referencing nor empirical 
testing. They reflect serious thinking about the matter, however, and 
in that sense may offer reasonably good leads for subsequent develop- 
ment. The primary reason for Including them in the present paper is 
to demonstrate that the field has in fact evolved vocabulary to 
describe subsets of RDD&E activities, and that these can be ordered 
in some reasonable way. 



constructs with which one is working, or the interrelationships between 
them. In the case of the treatment ascendant research the output is 
identification or clarification of the effects of certain treatment 
conditions on certain anticipated or observed outcomes. (These seem 
to represent functionally significant distinctions in that concept or 
construct clarification represents a markedly different kind of out- 
put than the effects of treatment conditions *) 

The second observation relative to Table 1 has to do with tne 
utilization of such a distinction. Given the nature of the outputs 
expected from these two kinds of rese'arch activity, and assuming that 
the nature of chat which is to he produced determines in large part 
what one has to do to produce it, it is probable that the two activ- 
ities require somewhat different strategies or somewhat different 
emphases :n their implementation. In relation to design, for example, 
the central issue in construct ascendant studies would be one of as- 
suring parameter te^-ting. In treatment ascendant studies the issue 
would be one of assuring clearly differentiated treatment conditions, 
control of all unwanted sources of variance, and selection of appro- 
priate dependent variables. Such differences necessarily have 
implications for the design and operation of research studies, as well 
as for the training of persons to carry them out. 20 

The third observation in relation to Table 1 is that it is possible, 
and probably desirable in the long-run, to extend the distinctions 
within research, development, diffusion , and evaluation to at least 
another level of differentiation. Table 2 suggests what such a dis- 
tinction might look like in the case of research. 21 It would seem 
unwise, however, to invest much energy in attempting differentiation 
at this level until first and second level distinctions have been 
f i rmly es t ab lis hed . 

Finally, by drawing attention to possible differences within research 
by distinguishing between classes of outputs rather than activities 
labeled “basic” and “applied,” it is hoped that progress can be made in 
eliminating the sense of second class citizenry that has so often ac- 
companied a focur on the “applied.” Since design and measurement re- 
quirements are just as rigorous for the drawing of generalizations 
around treatment conditions as they are around construct specifications , 



Obviou.ily, construct ascendant research also. has to deal with 
sources of variation , experimental treatments , etc. , and treatment 
ascendant research has to deal with the problem of construct parameter; 
but, the centrality or "ascendance" of these matters varies consider- 
ably in the two cases. It is this matter of centrality or ascendance, 
and the assumption of variation of operations that accompany it, that 
is highlighted by the proposed distinction. 

2 1 

The rationale for distinguishing between classes of research 
on the basis of the measurement properties of the variab les under in- 
vestigation rests on the observed variation in data collection, analysis, 
and interpretation strategies that link to these differences. 
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TABLE 2 



Illustrative Third Order Defiaitioas 



for Educational Research 


Construct Ascendant Research 


Treatment Ascendant Research 


' 

Variables limited 
to nominal or 
ordinal measure- 
ment 


Variables permit- 
ting interval or 
ratio measure- 
ment 


Variables limited 
to nominal or 
ordinal measure- 
ments 


Variables per- 
mitting interval 
or ratio measure- 
ments 



and theory development is equally critical to both, some of the devicive- 
ness that seems to follow the basic-applied distinction may be eliminated. 

Tables 3, 4, and 5 contain illustrative sets of categories that 
might ultimately refine into level 2 definitions for development, 
evaluation, and diffusion. In all cases the definitions appearing in 
these tables are intended to reflect the same level of discrimination 
as made in Tables 1 and 2. 

While it would be naive to assume that name changing in and 
of itself creates change in the way a construct is viewed, especially 
if it has assumed a special set of characteristics over a long period 
of time, it would be equally naive to assume that recasting a concept 
is without potential influence in this regard. If the recastings 
suggested in Tables 1 through 5 provide a reasonably sensible dif- 
ferentiation of differences within educational RDD&E, then it is 
possible that a major source of the variation in the operations needed 
to carry out such activities could be accounted for by classifying 
projects accordingly. Such at least must be the aim of the classi- 
ficatory framework if it is to have functional utility. 

Conceptual Set 2. Interactions 

Many problems that face education appear to be solvable through 
applying the strategies of research or development or evaluation or 
diffusion alone. Identifying factors that lead to reading disability, 
for example, or determining the effect of school lunch programs on 
pupil energy level, are problems that appear to be appropriate for 
solution through research. Problems such as the creation of curri- 
cular materials, assessment procedures and information management 
systems to support individualized instruction, or the creation of 
budgeting-costing systems that will serve effectively a school 
district’s efforts to manage by objectives, are problems that appear 
to be appropriate for solution through development. Problems such 
as getting all schools within a district to adopt a framework for 
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TABLE 3 



Illustrative Second and Third Order 
Definitions for Educational Evaluation 



Process Evaluation 



Product Evaluation 



Evaluative efforts which focus upon Evaluative efforts which focus 

the means by which a thing is done upon the consequences of that 



which is done 



Examples: Projects which attempt 

to assess the adequacy of teach- 
ing skills, or the appropriateness 
of children’s problem solving 
strategies 



Examples: Projects which are to 

determine the results of student 
learning in a newly devised curri- 
culum, or the worth of the re- 
sults of a research study 



Within the con- Within the con- 
text of a ’’forma- text of a **sum- 
tive** effort raative” effort 



Within the con- Within the con- 
text of a "forma- text of a "sum- 
tive" effort mative" effort 



establishing goals and objectives so as to assure their statement 
at approximately the same level of generality, or getting all 
institutions within a state that prepare student teachers to adopt 
a new basis for certification appear to be problems appropriate 
for solution through the strategies of diffusion. And problems such 
as determining the impact of a new curriculum upon the students, 
staff and parents of an experimental school, or in determining com- 
munity response to recent school board action, appear to be problems 
appropriate for solution through the strategies of evaluation. Indi- 
vidually and collectively, the solution of such problems contributes 
to the betterment of education; in the long run, however, education 
requires the solution of larger problems. 

An assumption underlying the proposed framework is that as 
problems become larger and more complex their solution requires the 
application of the strategies of RDD&E in combination rather than 
individually . 

In attacking problems of major scope in education, for example, 
creating the means for compensating early disadvantage In intellec- 
tual development, assuring that all children are able to read at a 
given level by a given age, or eliminating the basis for inschool 
dropouts, there seems to be little choice but to mount a coordinated 
effort that involves research, development, diffusion, and evaluation. 
The Right to Read Program launched by the U.S. Office of Education, 
the proposed Performance-Based Teacher Education Program, and the 
Sesame Street television series are cases in point. 
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TABLE 4 



Illustrative Second and Third Order 
Definitions for Educational Development 



Representational Development 


Operational Development 


Developmental efforts which focus 
upon the production of abstract 
or symbolic outputs. 


Developmental efforts which focus 
upon the production of concrete 
or manipulative outputs 


Examples: Projects which have 

as their end product a model, 
a set of hypotheses, a clas- 
sification scheme, or a cur- 
riculum outline. 


Examples: Projects which have as 

their end product a measurement 
system, an instructional pror.ram, 
an information management system, 
or a data processing machine. 


Products that 
are relatively 
small in scope, 
i.e., "module" 
development 


Products that 
are relatively 
large in scope, 
i.e., "system" 
development 


Products that 
are relatively 
small in scope, 
i.e. , "module" 
development 


Products that 
are relatively 
large in scope, 
i.e., "system" 
development 



TABLE 5 

Illustrative Second and Third Order 
Definitions for Educational Diffusion 



Interpretative Diffusion 


Adoptive Diffusion 


Diffusion efforts which focus 
upon the utilization of know- 
ledge and/or nformation 


Diffusion efforts which focus 
upon the utilization of tech- 
nology 


Examples: Projects which at- 

tempt to get teachers to in- 
clude career guidance infor- 
mation within the context 
of their courses , or utilize 
principles of reinforcement 
in the management of their 
classrooms 


Examples: Projects which attempt 

to get schools to adopt the IPI 
Mathematics curriculum, or a 
Program-Planning-Budgeting-System 
in support of education management 


Dissemination 


Incorporation 


Dissemination 


Incorporation 
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It also seems that even in attacking smaller problems all or 
some subset of RDD&E must be brought into play. Consider, for example, 
a school district that wishes to implement a reading program which 
(£) provides pupils a choice of specific learning objectives, (^) pro- 
vides for the continuous collection of information on individual child- 
ren as to their performance levels, interests, and history of success 
with given kinds of learning experiences, and (£) provides the oppor- 
tunity for pupils to move through the program at their own rate and 
in a preferred sequence. Suppose that upon review of available read- 
ing programs none were found that met these criteria and the district 
decided to develop such a program itself. Using the language of the 
definitions that appeared in the previous section of the paper, the 
experience of the district in preparing the program might be des- 
cribed as follows: 

Since the final output of the proposed effort is to be an 
individualized and data-dependent reading program, the ef- 
fort as a whole is developmental in nature. Enroute to pro- 
gram development, however, a variety of decisions need to 
be made; and, as in the case with all decision-makers, trust- 
worthy inf ormation - is desired in making them. 

As a consequence, formal data generating activities need 
to be undertaken. One type of data, for example, the attitudes 
of parents in the district and staff in the schools towards 
such a program, and the resources of the district that can be 
channeled into its development and operation, are of interest 
primarily to the district. According to the definitions 
provided earlier, the collection of such information should 
be considered as an evaluation activity, for there is little 
or no concern with its generalizability . There is concern with 
its reliability and validity, however, for without careful 
attention to such factors data simply are not trustworthy. 

Other data needed by a district in developing a program of 
this kind, and that would be of interest to other districts, 
are on the relationship between personality and background 
characteristics of children and their performance in a person- 
alized reading program, and the effectiveness of various materials 
presentation strategies for various kinds of pupils. Data 
such as these, at least if they are to be generalizable and 
thereby of optimal use to other school districts, represent 
outputs of research activity. A decision to make such data 
available requires that the principles of experimental design 
accompany the data collection efforts . As in the case of 
information to be used in support of on-line decision-making, 
it also requires that acceptable standards of reliability and 
validity be demonstrated. 

Once the program has evolved to the point of its being workable 
within one school, a decision might be reached to place it in 
several others. At this point ways have to be found to get 
staff members who have not had a part in the initial development 
of the program to be able to understand what it is about, be 
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willing to use it, be able to adapt it to their own instruction, 
and to become competent in its operation. After employing a 
wide variety of diffusion strategies — for example, workshops 
around the new program, a chance to observe the program in 
operation, and installation in a second school on a trial 
base — it was decided that formal training in its operation had 
to be undertaken prior to its implementation. To carry out 
such training, carefully designed training programs were 
developed , field tested ( evaluated ) , and then applied. 

Assuming the validity of the illustration, it is clear that even 
problems of a relatively modest scope require some combination of 
the strategies of RDD&E for their solution. 

The purpose of this section of the paper is to provide a frame- 
work for interactions between educational RDD&E as they are applied in 
real-world settings. In so doing it is proposed that two related 
interaction sets are needed to convey adequately the nature of these 
relationships: one which sees them as an interd ependent set of 

problem solving activities; and, one which sees them as an intrad ependent 
set of problem solving activities. 

1. RDD&E as a interdependent set of problem solving activities > 

As indicated in the Introduction to the paper and the example cited above, 
RDD&E seem to function as an interdependent set of problem solving ac- 
tivities — at least as they are brought to bear on problems of considerable 
scope. RDD&E as an INTERDEPENDENT SET of problem solving activities are 
defined as a situation-specific mix of research, development, diffusion , 
and/or evaluation activities that are coordinated over time so as to 
yield solution to a particular educational problem . Within such a 
view the particular combination of RDD&E activities relied upon in 
the solution of a particular problem, the particular intermeshing 
of those activities, and the particular sequence in which they are 
called upon, become situation-specific. Depending upon the nature of 
the problem being pursued RDD or E may be the initiating activity 
but, depending upon the strategy chosen for its pursuit, any of the 
other three activities may be called upon as follow-on or supporting 
activities. In the example cited, the school district could have 
decided to proceed with the development of the new curriculum before 
undertaking the related research and evaluation, or they could have 
reversed that order. And it may or may not have made any difference. 

The point is that while there is order and sequence to the use of 
RDD&E activities in the solution of a particular problem, the inclin- 
ation to impose a particular order and sequence across problem solutions 
is full of risk. It may be, of course, that in time one or more linear 
patterns may be identified as particularly successful in the solution of 
a particular class of problems, but that is something to be hoped for. 

It has not as yet been observed. 

Given such a view, the proposed framework is one which essentially 
incorporates the notion, of the decision-oriented and linkage models 
described by Gideonse . The commitment to outputs as a basis for defining 
RDD&E activities clearly puts the proposed framework in the camp of the 
decision-oriented model. The view of RDD&E as a problem-specific network 
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of interdependent activities also puts it in the camp of the linkage 
models described by Gideonse • The commitment to outputs as a basis for 
defining RDD&E activities clearly puts the proposed framework in the 
camp of the decision-oriented model. The view of RDD&E as a problem- 
specific network of interdependent activities also puts it in the camp 
of the linkage model, 

2 , RDD&E as a intradependent set of problem solving activities . 

In addition to viewing RDD&E as activities that can be called upon as a 
collectivity in the solution of large scale problems, or relied upon in- 
dividually in the solution of smaller problems, it seems also to be the 
case that RDD&E can function as a collective set of problem solving 
activities within even the smallest research , development , diffusion, or 
evaluat ion effort . Any research project , for example , has instruments 
to develop and evaluate , observers to train, and results to diffuse . Any 
development project has materials to prepare and evaluate , and if that 
which is to be developed is for general use within a district, a State 
or the Nation, then diffusion will also enter the picture. Similar in- 
tradependencies exist within diffusion and evaluation efforts, 

RDD&E as an INTRADEPENDENT SET of problem solving activites (RDD&E) 
are defined as a si tuat ion-specific mix of research , development, diffusion , 
and/or evaluation activities that are coordinated over time so as to yield 
sol ut ions to the particular problems within a particular RDD or E projec tT^ 2 
Within this framework, the outputs of rdd&e are the same as the outputs of 
RDD&E, except that they are usualJ.y smaller in scope and they are in 
service of a research or development or evaluation or c:if fusion activity, 
rather than an educational problem that extends beyond a single activity. 

Operationally the interaction of rdd&e activites within a given 
RDD or E effort appears to assume much the same form as RDD&E assumes 
in the solution of a large scale educational problem, that is, they form 
a situation-specific network. While the parent activity may set some 
parameter on the particular pattern of rdd&e utilized in its support 
(for example, research as a parent activity may call in different use 
patterns than development or evaluation), it is likely that variation 
in the patterning of an rdd&e network is a function of the focus of the 
RDD&E activity being pursued and the strategy chosen for its pursuit. 

Figure 1 illustrates schematically a network of rdd&e activities 
that could emerge in support of a research effort. As RDD&E are 
now being practiced in federally funded projects, a network of similar 
complexity would probably emerge within nearly all evaluation efforts; 
and, networks of considerably greater complexity would tend to emerge 
for development and diffusion efforts. 
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In the pages that follow whenever RDD&E is being discussed in 
terms of their intraproject dependencies they are identified as rdd&e. 
This notation is intended to facilitate the distinction between RDD&E 
as project activities that serve in the solution of large scale problems, 
i.e., problems that extend beyond a single project for their solution, 
and activities that serve in the solution of problems encountered 
within the conduct of a single project. 
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The Known Effectiveness of Instructional Treatment Q on Outcome A for 
Pupils Having x, y & z Characteristics 
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3> RDD&E as a nested set of problem solving activities . The view 
of educational RDD&E as a set of problem solving activities that can be (a) 
brought to bear collectively for the solution of large scale educational 
problems, applied individually to small scale problems^ or (c) applied 

collectively in service of any particular research, development j diffusion, 
or evaluation problem leads to a second order formulation; namely, RDD&E 
as a NESTED SET of problem solving activities. Conceptually it appears 
that large scale educational problems beget large scale RDD&E efforts, and 
that within such efforts there are multiple layers of RDD&E and rdd&e 
activities — all of which have the characteristics of a collective network 
about them. When viewed as interacting layers of activity, they also 
assume the additional characteristic of hierarchical ordering. The 
concept of a nested set of activities seems to capture these properties 
well. 



While helpful in conceptualizing the overall nature of educational 
RDD&E, the concept of nested sets provides little help in the operational 
problems one encounters in attempting to study such activities. One 
particularly difficult problem stemming directly from these properties 
is that of fixing the unit of observation, and the attendant problem of 
determining the decision rules to insure that that which one wishes to 
analyze can in fact be captured and manipulated. Put in other terms, 
the problem becomes one of identifying the unit of output that is of 
interest, identifying the boundary conditions for that output, and then 
placing that unit in the perspective of the larger output of which it is 
a part and the smaller outputs that comprise it. A set of decision rules 
that have been developed and used for this purpose are described in 
Volume V of the Oregon Studies. 

Conceptual Set 3. Mechanisms 

The first two conceptual sets that have been reviewed within the 
overall framework have dealt with the basic concepts of RDD&E and how 
they interact with one another in the solution of educational problems. 
Missing from the framework thus far is any mention of the organization 
and operation of such activities, and the resources needed to support 
them. Also missing from the discussions thus far is any reference to 
the mechanisms or vehicles that carry such activities. The purpose of 
Conceptual Set 3 is to provide a way of looking at such mechanisms. 

The purposes of Conceptual Sets 4 through 6 deal with the issues of the 
input, output, and operation of mechanisms; Conceptual Set 7 deals with 
the matter of their organization. 

The assumptions underlying the development of this particular con- 
ceptual set within the framework are the same as those' which underlie 
all of the others, namely, that a major source of variation in the 
operations required to carry out educational RDD&E is the nature of 
the mechanisms or vehicles that carry those activities, and that a 
framework designed to map the domain of educational RDD&E must be able 
to accommodate that source of variation. 
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By level . By and large the vehicle that carries the bulk of educational 
RDD&E in the Nation at this point in time is the project (Cook, 1971). 

PROJECT is defined as a formal or recognized unit of operation within 
which resources are brought to bear on and activities coordinated in 
th^e Solution of a given problem within a given time period. Sode fined, 
a project may be large or small, well or poorly financed, strongly 
or weakly staffed, or one or several foci, linked to or independent of 
related projects, housed within or outside of established institutions. 

All such factors are seen as potential sources of variation in the 
operation of RDD&E projects and are dealt with in further detail in the 
paragraphs which follow. 

Another vehicle that carries RDD&E activi ties is that of a program, 
with PROGRAM defined as a formal or recognized unit of operation within 
which resources are brought to bear on and activities coordinated in the 
pursuit of two or more projects that have a common focus . Within the 
present framework, it is proposed that two major classes of programs 
exist: those which are institutionalized and thoi^e which are project 

dependent. Institutionalized programs are those which have their resource 
base tied to a programmatic or goal-oriented effort rather than the 
specific projects which constitute it. Project dependent programs are 
those whose resource base is tied to the specific projects that comprise 
it. The program of work at the University of Wisconsin's Educational 
R&D Laboratory in cognitive learning, or in Harry Harlow's laboratory 
on the relationship between early learning experiences and later love 
relationships in primates are illustrative of institutionalized programs. 

The "programs" established inside most university departments, or in 
R&D centers that operate within the private sector, tend to be illus- 
trative of project dependent programs. Both kinds of programs can be 
found, of course, within university settings, public school settings. 

State departments of education, and the private sector. 



Cook cites a number of related definitions by Gaddis, Woodgate, 
and Robertson, respectively: 

". . .(an) organizational unit dedicated to the attainment of 
a goal — generally the successful completion of a development 
product on time within the budget, and in conformance with 
performance specifications." 

". . .work to be done or procedures to be followed in order 
to accomplish a particular (project) objective ." 

". . .the accomplishment of a number of actions in series and/or 
parallel in order to reach an objective (p. 4)." 

He also identifies four characteristics of projects which help differentiate 
them from most nonproject activities: they are usually finite in 

nature; they are usually complex in nature; they usually involve a series 
of tasks which relate only to that effort ; and, they generally consist of 
a once-through, nonrepetitive, or a one-of-a-kind activity. 
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Another vehicle which carries RDD&E activities is that of the in- 
stitution. INSTITUTION is defined as a formally constituted and recognized 
unit of operation within which two or more programs of R,D,D and/or E 
rest, and which assumes its primary identity from these programs . 

The federally supported education R&D centers and laboratories are 
cases in point. So are institutions such as The American Institutes 
for Research, Educational Testing Service, the Laboratory of Educa- 
tional Research at the University of Colorado, the Teaching Research 
Division of the Oregon State System of Higher Education, Systems De- 
velopment Corporation, and the Rand Corporation . 

As educational RDD&E is now moving, it is probable that projects 
will be housed increasingly within a program framework, and that pro- 
grams will be housed increasingly within specialized RDD and/or E 
institutions . 

In the present framework it is proposed that two factors associa- 
ted with project/program/institution definition account for much of the 
variation observed in such mechanisms; namely, their identity snd their 
level of complexity . 

Project identity . It is proposed that three factors contribute to 
the identity of any given project: (^) its focus , that is, whether it 

carries a research, development, diffusion, and/or evaluation emphasis: 

(^) the sett in ; in v/hich it rests, that is, whether it is within a 
school, a college, an R&D center, an educational laboratory, or a 
State department of education; and, (O its linkage to the setting 
in which it rests , that is, whether it is "appended” to its setting 
(in' the sense that it is external to the critical function and/or 
operation of the setting), or whether it is an integral part of its 
setting. The three-dimensional scheme that appears in Figure 2 is 
intended to illustrate the interaction of these factors as they in- 
fluence the identity of a particular project. The premise underlying 
the scheme is that operations will vary with project identity and that 
those who attempt to work within projects, train others to work within 
them, or study them need to be aware of that possibility so as to ac- 
commodate for the variation in operations that is likely to deiive 
therefrom. 

Project complexity . As in the case of project identity, the concept 
of project complexity is introduced as a means of organizing a set of 
factors that are assumed to be associated with variation in the opera- 
tions required to carry out any given project. Threa factors contribute 
to a project's complexity: (^) its resource base ; (^) the inter- 

program or interinstitutional alignments that are made or needed in 
its conduct; and (c) the output commicmencs that coerce it. The 
three-dimensional scheme that appears as Figure 3 1 intended to 
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To simplify language, the discussion of project/program/institution 
identity and complexity involve only the term project. It is anticipated, 
however, that these concepts are equally applicable to programs and insti- 
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FIG. 2. The dimensions of a project that contribute 
to its identity. 
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FIG. 3. The dimensions of a project that 
contribute to Its complexity. 
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illustrate the interaction of these factors. The premise underlying 
the scheme is that as the resource base of a project increases, as 
intra- and interinstitutional alignments increase, and as the range 
or magnitude of project commitments increase, the complexity of a project 
will increase. A further premise is that as project complexity increases, 
so too will the operations required to carry it out. 

Conceptual Set 4. Output s 

The conceptual sets thus far described provide analytic tools for 
mapping the "external'* features of the domain of educational RDD&E. The 
definition set provides the means for labeling a particular problem 
solving activity as research, development, evaluation, or diffusion; the 
interaction set provides the means for describing the relationship be- 
tween such activities; and, the mechanism set provides the means for 
describing vehicles which carry them. Conceptual Set 4 is the first 
of four sets that provide the means for mapping the "internal" features 
of RDD&E, that is, the organization and operation of mechanisms. 

Conceptual Set 4 focuses on the output side of RDD&E. Its purpose 
is to provide a generic set of descriptors that can be used in mapping 
the output structure of any research, development, diffusion, or 
evaluation project (or program or institution). The role of this set 
in the overall framework is pivotal, for it provides the necessary 
follow-through to the output definitions of RDD&E and it maps that 
which is probably the major source of variation within the operation 
of any particluar project. The premise on which this last statement 
rests is the relatively simple notion that what one needs to produce 
determines in large part what one does to produce it. 

A two-way classification scheme is used in mapping project out- 
puts: (^) a category set that identifies the major classes of outputs 

emerging from a project, and (lb) a category set that identifies the 
patterns or hierarchies of those outputs. Each of these is discussed 
separately in the paragraphs which follow. The section closes with a 
discussion of the standards held for outputs and their anticipated 
relationship to variation in project operation. 

Classes of outputs . The outputs of RDD&E projects are able to 
be sorted into one of three classes: product, event, or condition. 

PRODUCT is defined as a tangible or "hard" outcome of work effort 
that survives in a form that makes it transportable ; EVENT is defined 
as an outcome of work effort that results in the occurrence of an 
observable transaction ; and, CONDITION is defined as an outcome of 
work effort that creates a desired circumstance or environmental 
character is tic . Examples of products are reports, memos, drafts of 
reports, hardware, films, test items, and data-ga the ring instruments . 
Examples of events are conferences, formal announcements of management 
or policy decisions, speeches (as given, not as recorded on tape), 
seminars, and all types of transitory communication. Examples of 
conditions are circumstances such as parental involvement in a pre- 
school program, the achievement of learning outcomes on the part of 
children, and the creation of various working environments that sup- 
port the operation of a project. 
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The listing of outputs that are associated with a particular project 
is referred to as an OUTPUT INlTa, Table 6 contains such an index for 
a relatively small research project (per annum expenditure of less than 
$100,000) and Table 7 contains an output index for a moderately sized 
development project (per annum expenditure between $175,000 to $200,000)* 
Output indexes for evaluation and diffusion projects, as well as for 
larger research and development projects, appear in the case profiles 
prepared for the 20 projects described in the Oregon Studies *^^ 

Patterns of outputs * In addition to outputs differing in kind 
within a particular project, they also assume markedly different positions* 
Some outputs assume the position of a contractually targeted output* Others 
assume the position of being a component of a targeted output* Still 
others are simply facilitating to either targeted or component outputs* In 
all cases, however, every output that is associated with or emerges 
from a project can be placed in a patterned or hierarchically ordered net- 
work. 

Within the present conceptual set, a four level classification 
scheme has been used to identify the position held by each output 
identified within a particular project: a focal position, a terminal 

position, a component position, or a facilitating position* A FOCAL 
OUTPUT is defined as an outcome of work effort expected by contractua l 
obligation to emerge from a project ; TEEIMINAL OUTPUT is defined as an 
outcome of work effort that constitutes a focal output, an essential part 
of it, or a prerequisite to it ;^^ a C0^ff0NENT OUTPUT is defined as an 
outcome of work effort that constitutes an element of or one step in 
the approximation to a terminal output ; and a FACILITATING OUTPUT is 
defined as an outcome of work effort that is supportive to the gener- 
ation of focal, terminal, and/or component outputs, but is not in itself 
an instance of such outputs . Any output at any level may assume the 
form of either a product, an event, or a condition* 

The mapping of outputs associated with a particular project ac- 
cording to the positions they hold within it results in what has been 
termed an OUTPUT MAP. Each map depicts graphically the hierarchical and 
functional dependencies of outputs within c given project. Figures 4 
and 5 represent, respectively, the output maps for the projects whose 
outputs were indexed in Tables 6 and 7* As in the rase of output in- 
dexes, all 20 case profiles prepared within the Oregon Studies 
contain output maps. 



Relatively small projects were included for illustrative pur- 
poses in the present paper in deference to the length of output indexes* 
Output Indexes for large projects, that is, for projects with a funding 
base of over $500,000 per annum, have as many as 250 listings* 

Focal outputs are, therefore, one type of terminal output. 
Although terminal outputs which have not been specifically contracted 
for may also emerge from projects, the identification derived from 
empirical data, exemplified in Figures 4 and 5, shows only focal out- 
puts . 
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TABLE 6 



An Output Index for a Small 
Research Project® 



001 Final Report 

002 Description of Analysis Techniques 

003 Interpretative Report 

004 Computer Printouts 

005 Computer Programs 

006 Design of Statistical Treatments (2) 

007 Statement of Variables and Parameters (2) 

008 Literature Survey (2) 

009 Raw Data 

010 Theory Papers (2) 

011 Quarterly Progress Reports (4) 

012 Budget 

013 Financial Report 

014 Schedule*’ 



® A project that is being conducted by the Educational Testing Service. 
This is one project in a program of research that seeks to study the pat- 
terns and theories of stability and academic growth among student pop- 
ulations. The project describes academic growth and development of 
students using longitudinal data collected in a previous study. Dr. T. L. 
Hilton is serving as Project Director. 

Obviously, this list does not include all of the outputs that will 
emerge from the project. Initial, inhouse draft copies of the various 
documents cited are not included, nor are routine or incidental memos, 
informal planning sessions, etc. The criteria and decision rules 
governing the selection of outputs for inclusion in an output index for 
a particular project are reported in Volume V of the Oregon Studies. 
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TABLE 7 



An Output Index for a Moderately 
Sized Development Project^ 



Products : 



001 Final Catalog of Performance Objectives in English, Grades 9-12 

002 Commercially-published Edition of Performance Objectives in English, 

003 Final Report of the Project Grades 9-12 

004 Field Test Data from Use of Preliminary Catalog 

005 User Questionnaires 

006 Teacher Annotated Catalogs 

007 Interview Questionnaire (Guide to interviewers) 

008 Interviewer Written Reports 

009 Preliminary Catalog 

010 Refined (Conference 2) Approximation to Catalog 

011 Modified Field Test Plan 

012 Video Tape Critique of First Draft of Performance Objectives 

013 First Draft of Chapters for the Catalog 

014 Journal Report (Project Officer) 

015 Journal Report of Project Progress (Project Director) 

016 Dissenting Position Paper (Journal Article) 

017 Response to Dissenting Position Paper (Journal Article) 

018 First Draft of Introduction to Catalog 

019 Agenda for Initial Conference 

020 Outline of Catalog Goals, Objectives and its Organization & First Draft 

021 Examples of Performance Objectives 

022 Initial Statement of Goal Areas for English 

023 Initial Field Test Plan 

024 List of Field Test Sites 

025 List of Consultants (Confirmed) 

026 Proposal 



Events : 



027 Conference, Field Representatives 

028 Initial Work Conference (October) 

029 2nd Work Conference (March) 

030 Field Use of Preliminary Catalog 



A project that is being carried out jointly by staff from the University 
of Illinois, the University of Indiana, and Purdue University. It employs 
a strategy of development that involves a wide range of college and school 
personnel in the development process. Dr. J. N. Hook, University of Illinois, 
serves as Project Coordinator. 



235 



2-19 



Final Report 
P -001 



! — 1 


r 1 


r " 


— 


“T 

1 


1 


1 ^ 




* 03 




1 


1 iJ CN 1 


CO 


4 J m 1 


• fH 




1 


03 


1 -H 




1 c 


rH 


1 


1 ooo 1 




o o i 


1 T 3 


o 


1 




1 C 


a 1 , 


1 ^ 


1 




1 3 (X 1 


* CO 


03 04 1 


JC 


04 


1 


DQ , 


1 ^ 


^ 1 








< 1 


1 *H 


1 


1 ^ 




1 














L 1 

__! 


L«. 


Ll±_ 




l: 


J 






CO 




4 J 








o 




a 


rH 


03 


u 


1 X 5 rH 


03 


rH 


U 


CO o 


U 


CO 1 


CO 


03 Ot 


3 


U 


O* 


GO 




O 




u 




04 





CO O 


u 


M rH 


o 


03 O 


03 


a 1 


JC 


CO 04 


H 


04 





0 ) 








> 




CO 




•H 




M 4 J 




4 J 4 J m 




03 3 




03 )H o 




4 J O <f 




MOO 








a a 1 




pj w 

a c o 




U (D 04 




e 1 




03 




O U 04 




4 J 




CJ Ol 




c 








M 







■ - 1 
03 CTk, 
4 J O ' 

5?| 

L__ J^j 



3 



\ 



M Vi ' 

03 B I 

4J CO * 

L_ a u a i 

n Q- OOO I 

BO! 

o M i 

O 04 

I. 1 



I 



I 



w 

4 J 

c 

03 

B 



n 1 “ “rn“ n 



• I 



i CO ■ 

, M-l 03 I 

I O M I 

H VO, 

1C ol 
. OOiH o. 

I -H CO I I 

W O Ol 

^ 03 *H 

I Q ^ 

I -H I 

. CO I 

^ I 

L_ JP- J 




■"VI 


u 

03 


^ 1 


4 J 


c , 


03 


03 1 


B 


B 1 


cO 


^ 1 


4 H M 


4 H CO * 


O CO 


O 03 1 


04 


U v£) 


4 J 


C H O — 


C *0 o 


tJ 3 O 


03 3 O 


*H rH I 


B CO 1 


CO CO CM 1 


03 04 


03 U 1 


4 J (0 


Q -H . 


CO 03 


4 J 1 


4 J rW 




C /3 ,0 


•H 1 


CO 


iJ 


•H 


CO j 


M 


Li ' 


CO 


J 


> 



W 

V* 

03 

4 J 

03 

B 

M-l CO 



I 

4J 04 

C 1^1 

03 -O O * 

I e c o 
S CO M 

4 J Ol - 

CO CO 
U 03 

I 

' s ' 

L.t J 



I n 

I I 



03 

u 

c 

I c 

4 dj 



I I 



I 

03 00 . 

> O I 

u o , 

=» * I 



U tA 04 

•H 
H-J 



I 



^ I 

M I 

c >. , 

4J 03 00 
CO ?> O , 
U U <D\ 
03 3 I * 

H ^ ^ I 

I 



CO 

4J 

c 

a 

c/3 4J 
4J 3 00 
3 o C 
Cl *H 
4J <U <U 
C CO 
03 4J 
C -H 
O rH 
cO CX T-l 

e U 
O <0 



3 

o 



u 

o 



CO 

4J 

3 

a 

4J 

3 

O 



fjt 4 CJ 



no 

C 

03 

OC 

0/ 



jc 

CJ 

u 

CO 

03 

CO 

03 

U 



cO 

B v £3 

CO 



u 

o 

4-4 



*n 

CO 

H 



03 
Cu 03 
cO CO 

B ^ 

4J 4J 
3 CJ 
a 03 

4J *r-j 

3 O 
O U 
CL 



O 

M 

04 



o 

ERIC 



236 






o 



* 




(March Conference- ^ 



I Video Tape Critique 

I L-012 






p . . -1 


Thirst Draft Introduction 


1 


P-018 



iJournal Report by^ 

I Project Officer [ 1 

I P-014 ! I 





I Journal Report of I i 
(Progress by Principal*^' 
^^nvest^p.ator __ 



First Draft Chapters for the Catalog 
of Objectives P-013 

*Virst Approx, to Catalog: Goals, Ob-^ ^J^esponse to Dissentinpi 

Jectives, Guidelines to Organization , i jj’osition Paner. P-017j 
I (output of Conference) P-020 

(October Conference! 



L _ 



E-028 



C' “ “ - C““Z ^ 
(Conference Agenda 

I P-019 I 



J 



i_ Dissenting Position, 
i_Paper _ _ _P-016 , 



^Examples of I’erfonnance Objective^ 

(for each Goal Area P-02l| 

- - - 



Initial Goal Statement 
P-022 



^ist of Confirmed 
, Consultants , 

L - _ _l 



_l Proposal 



P-026 



*Field Test Plan* 

■' P-023 r 

I ' 



J^^ist of Field Test ' 

^ Sites P-02^i ^ 



Legend : 

P ■ Product Focal output 

E ■ Event Component output 

----- - Facilitating output 



FIG. 5. An output map for a 
moderately sized development 
projft^i^ee Table 7). 



237 



Output Standards . In addition to outputs varying as to kind and 
position held, they also vary as to the standards they are produced 
against. STANDARDS are defined as the criteria applied to, or level 
of excellence expected of» a particular output . As such, standards may 
be explicit — in the form of criterial attributes and rules of evidence 
pertaining to the demonstration of those attributes — or they may be 
implicit. Whether explicit or implicit, a working assumption of the 
framework is that the standards set for a particular output influence 
significantly the nature of the operations required to produce the 
output. As such they become a significant source of variation in pro- 
ject operations and thus must assume a place of prominence in the overall 
framework . 

Contrary to other potential sources of variation that have been 
dealt with in the framework, a category set has not been established 
to differentiate types of output standards. A good deal of data have 
been collected within the Oregon Studies about standards, and descriptive 
category sets have been established for their first order coding. But, 
as yet, no logical structure has emerged from these descriptive codes 
that permit their classification in terms of higher order abstractions. 
This is not to say that such a code is unneeded or undesired. It 
simply reflects the incomplete state of development of the framework. 

Because of the potentially powerful role that standards play in 
the design and implementation of RDD&E activities, a word needs to be 
said about our efforts to collect data on them. Of all the information 
sought in the Oregon Studies, that which focused on standards was the 
most difficult to obtain. In the few projects where standards had been 
made explicit such information was easily collected. Also, in cases 
where standards were implicit but generally understood, persons inter- 
viewed were able to state them readily. In most cases, however, 
interviewees found it difficult to make public both the standards they 
were operating upon and the indicators they looked to as evidence of 
the realization of those standards. Because of the assumed relation- 
ship between standards and project operations, and because of the in- 
timate relationship between standards and project impact, it would 
appear that one of the simplest yet most far-reaching ways of improving 
the impact of RDD&E in education would be to insist that output stand- 
ards for at least the focal outputs of a project be made explicit, that 
the indicators acceptab le as evidence of the realization of those stand- 
ards also made explicit, and that such evidence be demonstrated. 

Conceptual Set 5. Operations 

Conceptual Set 4 provided descriptors of the outputs associated 
with a particular project. Conceptual Set 5 is intended to provide 
a set of tools for describing what it takes to produce those outputs. 

This conceputal set assumes an extrememly critical role in the overall 




Over 100 interviewees were queried in the Oregon Studies as to 
the standards they held for over 250 outputs. These data, and the first 
order category sets used to code them, appear in Volumes I and IV report- 
ing the Oregon Studies. The decision rules governing their collection 
appear in Volume V. 
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framework, for it describes the work that is done wichin the context 
of RDD&E activities. 

As in the case of project outputs, a four level classificatory 
structure is used to describe operations. These include category 
sets that describe (^) the major functions that need to be executed 
within any project, (_b) the activities engaged in to carry out 
those functions, (O the tasks pursued to carry out a particular ac- 
tivity, and (^) the actions involved in carrying out a particular 
task. Operationally, these sets of categories are interdependent and 
hierarchially ordered and provide, thereby, a description of project 
operations that varies from a gross to a minute level of detail. They 
also represent category sets that **do double duty" in the sense thac 
they are used to describe project operations as a whole as well as 
operations pursued in the generation of a particular output. This 
dual purpose of the category sets is discussed in some detail in 
the paragraphs which follow. 

Functions . Drawing upon the experience of industry, three sep- 
arate though mutually supporting functions are proposed as operating 
within all projects : production, management, and policy setting. 

POLICY SETTING is defined as the establishment of standards for the 
focal output (s) of a project^ the allocation of resources toward the 
realization of that output, and the establishment of guidelines as to 
the desired procedures to be followed in and anticipated consequences 
to derive from the project as a whole * The function of MANAGEMENT is 
defined as the orchestration of the energy, time, materials, facilities , 
and information available to a project so as to optimize desired outcomes 
and minimize undesired outcomes . The function of PRODUCTION is defined 
as the execution of the operations required to achieve a planned out- 
come of a project, given the constraints and facilitators provided by 
policy and management . The relationship of these three functions to 
the outputs of a particular project are shown schematically in Figures 
6 and 7. The classification of the outputs identified in the two pro- 
jects that have been used for illustrative purposes by the function they 
serve is summarized in Tables 8 and 9. The decision rules governing 
their application appear in Volume V reporting the Oregon Studies. 

Three observations need to be made in relation to this particular 
category scheme. First, as seen in Figure 7, outputs can be mapped 
in terms of their production, management, and policy-setting functions. 
Operationally, this requires a refinement of the output maps illustra- 
ted in Figures 4 and 5 by ordering them according to the schema outlined 
in Figure 7. By so doing, a map of finer detail is provided. 

The second observation made is one that is fairly obvious by 
riow. Outputs receive classifications beyond those given them by 
the category set specifically designed to describe them, i.e., Con- 
ceptual Set 4. Classification as to the function they serve is a 
case in point; so, too, is their classification according to the RDD&E- 
rdd&e taxonomy, i.e., Conceptual Sets 1 and 2 . 

The third observation is that policy and management functions do not 
seem to stand high in the awareness of project staff; but, to the 
extent they do, awareness seems to be in direct proportion to project 
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size. On the basis of the experience gained in the Oregon Studies it 
is also our impression that educational RDD&E personnel are relatively 
naive about the matters of policy and management 

Activities, tasks, and actions . The category set describing func- 
tions is intended to order in a gross way the operations that need 
to be carried out for either a project to be completed or an output to 
be produced. Three additional category sets are employed within the 
conceptual framework to describe in detail the operations required to 
produce specific outputs: one that describes the activities involved, 

one that describes the tasks involved, and one that describes the 
actions involved. ACTIVITIES are defined as the largest meaningful 
units of work engaged in to carry out a particular function ; TASKS are 
defined as the largest meaningful units of work engaged in to carry 
out ai~i activity ; and ACTIONS are defined as the largest meaningful 
unit of work engaged in to carry out a task . 



Focal Output 



The Policy- 
setting 
Function 



▲ 




The 

Production 

Function 



The 

Management 

Function 



▲ 




FIG. 6. The relationships among policy-setting, management, 
and production functions with reference to the focal 
output of a project. 



Desmond Cookes recent book Educational Project Management provides 
a good deal of information about the "nuts & bolts" of project management 
(Cook, 1971). He does not deal, however, with major conceptual or theo- 
retical issues relating management to policy and production, or combinations 
thereof. For references that do attempt to deal with such matters, see 
Adams (I960), Argyrus (1960), Etzioni (1960), Griffiths (1964), Cockcroft 
(1965), and Hemphill (1970). 
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FIG. 7. The relationship among focal, terminal, component, and facilitating 
outputs as they are positioned in service of policy-setting, man- 
agement , and production functions . 



TABLE 8 



Outputs of a Stoall Research 
Project Ordered by the Function 
Each Output Serves® 



Outputs Serving the Policy Function 
(none noted) 



Outputs Serving the Production Function 


001 


Final Report 


002 


Description of Analysis Techniques 


003 


Interpretative Report 


004 


Computer Printouts 


005 


Computer Programs 


006 


Design of Statistical Treatments 


007 


Statement of Variables and Parameters 


008 


Literature Survey 


009 


Raw Data 


010 


Theory Papers 



Outputs 1 


Serving the Management 


Function 


oil 


Quarterly Progress 


Reports 


012 


Budget 




013 


Financial Report 




014 


Schedule 





O 




a 



See Table 6. 



TABLE 9 



Outputs of a Moderately Sized Development 
Project Ordered by the Function 
Each Output Serves^ 



Outputs Serving the Policy Function 

014 Journal Report (Project Officer) 



Outputs Serving the Management Function 

003 Final Report 

004 Field Test Data from Use of Preliminary Catalog 
006 Teacher Annotated Catalogs 

008 Interviewer Written Reports 

011 Modified Field Test Plan 

012 Video Tape Critique of First Draft of Performance Objectives 

015 Journal Report of Project Progress (Project Director) 

016 Dissenting Position Paper (Journal Article) 

017 Response to Dissenting Position Paper (Journal Article) 

019 Agenda for Initial Conference 

023 Initial Field Test Plan 

024 List of Field Test Sites 

025 List of Consultants (Confirmed) 

026 Proposal 



Outputs Serving the Production Function 

001 Final Catalog of Performance Objectives in English, Grades 9-12 

002 Commercially Published Edition of Performance Objectives 

in English, Grades 9-12 
005 User Questionnaire 

007 Interview Questionnaire (Guide to interviewers) 

009 Preliminary Catalog 

010 Refined (Conference 2) Approximation to Catalog 

013 First Draft of Chapters to Catalog 

018 First Draft of Introduction to Catalog 

020 Outline of Catalog: Goals, Objectives, and its Organization 

and First Draft 

021 Examples of Performance Objectives 

022 Initial Statement of Goal Areas for English 

027 Conference, Field Representative 

028 Initial Work Conference (October) 

029 2nd Work Conference (March) 

030 Field Use of Preliminary Catalog 
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See Table 7. 
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, 11 »- 'iliMply lalx’ln llwi» ja.Tl/jln Lo liM.ri^aN 1 mkIV rollni'd r>r detailed de- 
r 1 jil 1 niih nl Work |K‘rlori(UMj by project btal I m jirodut.e Hpeclfled out- 
puLu. The have no absolute referrnts as to slice or subscance slnct^ 
they are tied to outputs, and outputs vary endlessly on both counts. 

As a consequence, the boundary conditions that define a particular 
level of operation are relatively flexible and open to influence from 
project (output) characteristics. The decision rules governing ap- 
plication of the category set are also reported in Volume V of the 
Oregon Studies. 

As in the case of output standards, formal category sets that 
differentiate classes of activities or tasks have not as yet been 
established within the framework. Data have been collected on the 
activities and tasks that people have engaged in to produce over 350 
outputs, and these data have been coded in first order descriptive 
category sets. However, logical structures that would permit the 
description of tasks and activities at a higher order of abstraction have 
not been developed from these codes. The descriptive categories used 
in the Oregon Studies to code the task and activity data are described 
in Volumn V of the Oregon Series on RDD&ii. The data collected on 
tasks and activities, by output, appear in Volume I, 

Conceptual Set 6. Inputs 

Using the standard input-operation-output language of systems analysis, 
conceptual sets have been provided thus far for the description of the 
output and operation dimensions of projects. The purpose of Conceptual 
Set 6 is to draw attention to the input side of the equation. The term 
**draw attention to" is used advisedly in this case for little has been 
done thus far in the development of the framework to conceptualize the 
nature of project inputs. As a consequence, this section of the 
paper is primarily intended to alert the reader (^) that project input 
represents a critical dimension of the RDD&E process, (^) that it can 
be mapped in much the same way that the output and operations dimensions 
can be mapped, and (jc) that because of its criticalness, the conceptual 
tools needed for its mapping should be created. The following observations 
are intended only as suggestions as to what some of the critical di- 
mensions of project input might be. 

Broadly speaking project input can be equated with the resource 
base available to a project. From one aspect, resource availability can 
be equated with dollars, for it is with dollars that time, personnel, 
and support systems are purchased. From another aspect, however, re- 
source availability goes much beyond dollars and cents. In some 
cases time and talent can be purchased and in others they cannot. Some- 
times it is possible to gain access to the support systems needed for 
project ')peration (for example, high speed or large-storage computer 
capability, on-call access to subjects of field test populations, im- 
mediate tum-around time for reproduction or printing), and' sometimes 
it is not. From this point of view the nature of resource accessibility, 
and use beco*»es a critical dimension of project operation. 
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Another dimension of input to project operation, and the only 
dimension with which the framework currently deals, is the matter of the 
knowledges, skills, cind sensitivities that project personnel bring to 
their tasks. In some respects this may be the most critical of all 
project inputs, for its relationship to project operation, and thus 
project output, is essentially a direct one. Knowledges, skills, and 
sensitivities have been conceptualized as ENABLERS of work performance, 
and, therefore, defined as prerequisites to the performance of the 
funct lons/actlvlties/tasks/actlons that are required to produce a par- 
ticular output . Operational definitions, decision rules, etc. governing 
the application of this category set are reported in Volume V of the 
Oregon Series. Volumes I and IV report the data collected through their 
application. As in the case of output standards and descriptors of 
work (activities, tasks, actions), no category sets have as yet been 
developed that differentiate within classes of enablers . 

Much has yet to be done in the conceptualization of project inputs. 

With a concept as complex as this, there are undoubtedly a wide range 
of variables to be dealt with, and a wide range of options as to how 
best to deal with them. The purpose of this conceptual set is to sen- 
sitize the reader to the criticalness of the input dimension of project 
operation and encourage that it be addressed systematically and in 
some detail. Once more fully conceptualized, it needs to be added to 
the other category sets that comprise the framework and applied to the 
overall task of mapping the internal workings of the domain of educational 
RDD&E. 

Conceptual Set 7, Organization 

One other source of anticipated variation within the domain of educa- 
tional KDD&E is the manner in which projects/programs are organized for 
work. To do that which they must do, projects (^) organize their resources, 
and (^) organize their work into manageable units. How they go about 
doing this varies a great deal. And, it may well be that how they go about 
doing it affects in a significant way both the nature of project outputs 
and how those outputs get produced. 

This final conceptual set in the present framework provides for the 
description of the manner in which projects are organized. As in 
the case of Conceptual Set 6, however. Conceptual Set 7 is intended 
only to **draw attention to" the need co consider project organization. 

As a point of departure, it appears that nearly all RDD&E projects 
are organized around three dimensions : a policy dimension, an administra- 

tive or management dimension, and a work responsibility or job dimension. 



These parallel the policy-setting, management, and production 
functions elaborated under Conceptual Set 5# As dealt with there they 
served essentially as first-level organizers for viewing what people 
do to effect project output. Here they are intended to serve as guides 
for viewing how people or g anize themselves an d their work to effect 
project output. 
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Moreover, it appears that all three dimensions vary as a function of a 
pr'^ject size, the nature of focus of a project, and the particular style 
preferences of the person or persons responsible for project management 
and/or policy-setting. Some projects, for example, establish formal organ- 
izational units for purposes of administration and policymaking, whereas 
others operate with very informal units. In some cases such units are 
purposefully kept small while in others they are allowed to grow large. 

And the terminology for such units seems to vary as widely as the organ- 
izational structures themselves . Labels such as boards , councils , and 
commissions in the case of policy-setting, and divisions , departments , 
units, areas, and sections in the case of administration seem to be ap- 
plied interchangeably. The one constant that seems to exist with respect 
to organization for administration and policy making is that, as projects 
increase in size, differentiated structures to handle these matters seem 
to emerge. In a one-man project the need for such differentiation is 
minimal; in a 50-man project it is a necessity (Cockcroft, 1965). 

it' same diversity seems to hold with respect to job definitions. 

With the exception of the job of Project or Program Director, jobs 
seem to be defined primarily in response to the peculiar demands of 
a particular project or the style of those responsible for its manage- 
ment. Jobs seem to be defined differently by different organizations, 
and, within a single organization, differently for different projects. 

It is also common for jobs to change definition several times within 
the life of a project. As a consequence, jobs seem to be even less 
stable in their definition than units of administration and policy making. 
Job definitions nevertheless form a significant organizational frame- 
work within a project, and it would seem essential that they be included 
in a complete mapping of the domain of educational RDD&E.'^ 

This completes the description of the various conceptual sets that 
make up the proposed framework for the analysis and empirical investi- 
gation of educational RDD&E. In recap form, the first three of the 
seven sets are designed to map the external parameters of that domain; 
the last four its internal parameters. It is the thesis of the paper 
that in combination these sets are exhaustive, that is, they provide 
the conceptual tools needed to map sensitively the entire domain. 

This is a thesis to be tested, however, and it is the view of the 
authors that by testing it the tools of educational RDD&E will be 
sharpened significantly. Thus construed, the framework must be 
viewed as provisional, inviting of empirical test. The rationale 
underlying such a view is outlined in the Introduction to the paper. 



“ The relative instability of job definitions within RDD&E makes 
them unsuitable as a unit of analysis for performance operations. As 
a consequence, in the Oregon Studies, operations data and data on the 
knowledges, skills, and sensitivities needed to perform those operations 
were collected against specific outputs rather than job definitions. The 
rationale for this strategy was straightforward J outputs are tangible, 
relatively stable entities against which work can be described and per- 
formance assessed. While subject to project-specific variation, an output, 
or a class of outputs, is likely to serve a larger number of projects. As 
a consequence they appear to provide better (more stable, generalizeable) 
units of observation and analysis than do job definitions. 
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The Parable of the Roads 



"Would you tell me, please, which way 
I ought to go from here?*' (asked Alice.) 

"That depends a good deal on where you 
want to get to," said the Cat. 



The exchange between Alice and the Cat is most fitting as an 
introduction to the last section of the paper, for in the writers* 
judgment, educational RDD&E is at a cross roads. From our view there 
are three reasonably well defined roads which can be followed, though 
none promises a particularly smooth ride. 

The first is the road that has characterized educational RDD&E 
historically. To a large extent this is a road dominated by clas- 
sical research design and classical research interests in education, 
for example, acquisition and retention *=tudies, transfer studies, 
teacher characteristics studies, and teacher behavior studies. Eval- 
uation, development, and diffusion, as these concepts have been discus- 
sed in the present paper, are largely missing. In the light of the 
present day this is a relatively narrow road and one that is marred by 
potholes, blind corners, and an almost nonexistent sign post system 
alerting the traveler as to where he is going. 

The second road that is open to travel by educational PDD&E per- 
sonnel is one that has been opened only recently and one that is 
not as yet completed. This is the road that takes the form of a super- 
highway begun less than a decade ago and has lanes accommodating not - 
only research, but development, diffusion, and evaluation, as well. 

And, each lane of the superhighway is sufficiently wide and well- 
surfaced so as to accommodate new, high speed, high powered, roomy 
vehicles. Exemplary efforts along these new lanes are such endeavors 
as "county agents" for encouraging and demonstrating innovative 
educational methodologies, the vast ERIC dissemination s>stem, educa- 
tional accountability and performance contracts, systematic derivation 
and structuring of relevant student performance objectives, and develop- 
ment of major instructional packages for increased management of 
the total learning situation for individual students. The problem 
with this road is that it is still under construction; it has not 
as yet been surfaced; and, only the first few miles have been es- 
tablished. As a consequence there are signs all along the way that 
read "Proceed with Caution," "Work in Progress,** **Detour,** and 
**Stop! New Road Under Construction." There are sign posts along the 
way that tell the traveler where the road has come from, and where it 
ultimately hopes to go, but confidence in the likelihood that it 
will even reach its destination is a matter of faith. 
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Tlie ihird road discernible at the cross roads is an access 
road tliat parallels the freeway. Its ultimate purpose is to provide 
thi* supj>orl systems needed to plot the co\irse ol tiie freeway, build it, 
iii.iinLaJn it, service its travelers, and develop new capabilities at road 
buLJ.dini'. It Leads to the tool shops, the research laboratories, and 
the testing centers of the road builders, it also leads to centers 
that prepare personnel to man the roads, or to regenerate or re- 
freshen those who have been on the road for a long while. 

Unfortunately for the traveler, this is a newly commissioned road, 
and thus far only its pathway has been marked. The traveler is not 
at all sure where the road leads or what the tool shops, research lab- 
oratories, testing stations, or personnel training centers will look 
like. Moreover, there is no assurance that the road will ever be 
opened, or that the support s*'rvices it is supposed to give access to 
will ever be established. The keepers of the roads are not at all 
sure that an access road, and the support systems it is designed to 
provide, represent good investments. Alter all, the concept of the 
freeway emerged without such "fancy extras;" and, it’s even being built with- 
out their assistance. Why should precious resources be spent on that 
which isn’t absolutely necessary, and which may not be needed at all? 

l^ien standing at the cross roads which of the three options 
available should the RDD&E traveler pursue? Should he take the well 
traveled road of research, taking particular care to avoid the potholes, 
approach the blind curves with caution, and take extra care to be sure 
that the road is leading him to the end he wishes to reach? Or, should 
he enter the superhighway where, even with all the "Proceed With Caution" 
and "Road Under Construction" signs, the other travelers are pressing 
their new machines at such a pace that there is danger to those who have 
a smaller machine, or who wish to drive more slowly? Even with the 
danger of being run over or shoved out of the way and with all the 
noise, the dust, and the confusion of the big machines moving down all 
four lanes, the superhighway still has some attractions. It looks like 
it’s going in the direction a lot of RDD&E personnel wish to go, and it 
looks like it would let one get there relatively fast. And, once the 
road is surfaced, it should be relatively smooth traveling. Such 
attractions are hard to turn down. 



The commission for the access road and some of its support 
systems came less than two years ago with the decision on the part of 
the Office of Personnel Development and Training within the National 
Center for Educational Research and Development, U.S. Office of Educa- 
tion, to establish an empirical base for their training programs. Some 
advance survey parties such as Clark and Hopkins (1969), Sieber and 
Lazarsfeld (1966), and Cronbach and Suppes (1969), had studied the advis- 
ability and feasibility of creating such a road. But, it was not until 
the funding of the AERA Task Force Studies, the Oregon Studies, and the 
Training Design Projects that well financed survey parties and planning 
studies were established. The results of the Training Design Studies 
are in and three personnel training centers have been established. The 
results of the survey parties are only now being completed so the future 
of the access road is still uncertain. 



What are the attractions of the road that leads to the information 
and technology base that permi ts the improvement of the first two 
roads or the building of better roads in the future? It leads in a 
sense to the source of that which makes more hopeful the prospects of 
traveling either of the other two roads. All that equipment looks 
impressive — it should help~-but too few have used it and many lack con- 
fidence to climb into the operator’s seat. 

So what road should the RDD&E traveler choose? Fortunately, 
while one man has to make one choice, the discipline of educational 
RDD&E can choose all three, and pursue them simultaneously « It is 
the writers’ belief that this is the strategy which should be followed . 
The rationale underlying this position, some recommendations as to 
steps to take along each way, and some implications of the present paper 
for those steps are outlined in the paragraphs which follow. The paper 
closes with some general comments on the applicability of the present 
paper to personnel training and policy setting. 

Following an Improved Old Road 

The case has been built sufficiently well and sufficiently often 
for the necessity of a solid knowledge base for any of man’s endeavors, 
so that argument need not be repeated here. Wliat does seem to be worth 
saying is that the conceptual frameworks that support or underlie the 
development of a strong knowledge base must be kept in order. The con- 
structs or concepts that constitute those frameworks must be well expli- 
cated and critically tested for their utility. And, advances in design 
and methodology must keep pace with the advances in theory so that the 
power of such frameworks can be fully utilized.. 

Fortunately, the kind of conceptual base being called for is 
beginning to emerge in education. The work on aptitude treatment 
interaction, the concept of niathemegenic research (Rothkopf 1970 ; 

Frase, 1970), the concept of learning environments (Barker, 1968; 
Mitchell, 1969), and the concept of a teacher as the manager of 
learning environments (Tyler, 1950; Schalock, Kersh, and Horyna, 1970a; 
Schalock, 1971) are all indicative of the kind of change envisioned if 
research is to provide the knowledge base that has utility to other 
researchers or to practitioners in the field. 

Joining with the Builders on the New Road 

Given the need for, the power of, and the demonstrated potential 
of the tools of systematic development, evaluation, and diffusion, there 
would seem to be no other choice but to join forces with the engineers 
of these activities. New ground is being broken in each of these areas, 
some wisdom is accumulating, methodologies are being devised and tested, 
and enough outputs are emerging from them that have enough value to 
enough people that a wide-spread base of support and enthusiasm for them 
is being engendered. As pointed out in the present paper, however, 
there is sufficient confusion and uncertainty in all three of these 
areas that their effectiveness at this point in time is limited. It is 
also probable that it will continue to be limited until our knowledge 
about these activities is significantly increased. A major purpcjse 
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of the present paper was to provide a conceptual framework that would 
facilitate the extension of that knowledge base. 

Building on the Work of the Advance Survey Parties 

In the writers’ view the critical road to be followed in the immed- 
iate future is the road that leads to the increased understanding of the 
domain of educational RDD&E. The case has been argued in the present 
paper that the activities that comprise this domain constitute the primary 
problem solving tools available to the field of education, and that to 
employ them well they must be more fully understood than they are at presen 
It is also argued that a necessary way to increase our understanding 
about such tools is to submit them to empirical investigation. The major 
thrust of the present paper is a conceptual framework that defines and 
relates the conceptual structure of the domain in such a way that it 
could become subject to empirical investigation. Given the availability 
of a framework (assuming that it is adequate to the task) , the critical 
need is for its empirical referencing. A great deal of time should not 
be wasted before building upon the work of the advance survey parties that 
is now being reported. 

Personnel Training Centers 

The context within which the present investigations were commissioned 
was one of personnel training. Largely because of dissatisfactions with 
the old road and the manpower needs of the builders of the superhighway, 
the move to obtain empirical data on the domain of educational RDD&E was 
undertaken. New training programs have been established^^ and are ob- 
stensibly awaiting the data from the survey parties to help guide the 
substantive development of their programs. The question that must now 
be considered is one of the utility of the data that have been collected. 

Without question, the data from the empirical studies will have 
some utility for the new and existing training programs for educational 
RDD&E manpower. While speaking now only for the Oregon Studies, data 
will be forthcoming on all dimensions of the conceptual sets provided 
in the previous section of the paper. While these data are primarily 
descriptive and parameter testing, rather than normative or hypothesis 
testing, they will do much to identify the nature of RDD&E activities 
as they appear within education at this point in time. As such, this 
should provide necessary sensitivities to the designers of training pro- 
grams, even though they will not provide definitive answers to sub- 
stantive questions about the field. This is not an unexpected circum- 
stance, for the decision was made early in the Oregon Studies to sacri- 
fice data yield for methodological development.^ 



Three experimental training programs have been established as of 
this writing; The Far West Laboratory Consortium which focuses on the 
preparation of developers and diffusers; the Learning Research and Devel- 
opment Center in Pittsburgh which focuses on the preparation of instruc- 
tional systems designers; and the Consortium of the Ohio State University 
which focuses upon the preparation of field-centered evaluators. 
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A number of training, staffing, and organizational issues were 
raised during the course of the Oregon Studies Project. Volume 1 presents 
these issues in light of the data which have been collected and analyzed. 



From the writers' view this situation is not disappointing. The 
Oregon Studies alone have generated considerable amounts of new data 
about the nature of RDD&E activities, more than can be accommodated 
fully by a training program in its initial year or two of operation. 
Hopefully, additional studies will soon be undertaken so that, as the 
data generated by the present studies are internalized, the next level 
of data will become available. This assumes, of course, that there will 
be some who will travel the access road that leads to empirical research 
and development with respect to educational RDD&E, a hope that is contin- 
gent upon commitments of the keepers of the road . 

Notes to the Keepers of the Roads 

Like the personnel who staff RDD&E activities, those who determine 
policy about such matters and control funds that support such activities 
have come to the same crossroads. And, as in the case of RDD&E personnel, 
it is the writers' belief that they should support work on all three 
roads. The basis for this belief is not the simple, blind assumption 
that good things will come from the indiscriminate support of all kinds 
of activities. It rests instead on the kinds of arguments that have been 
outlined in the paragraphs above, and the belief that the essentially 
yet-untraveled road represents a significant new option for policy-makers 
to support. 

It is also our belief that the new option will be an attractive one 
at the policy-setting level, for in it there is promise of sharpening 
the tools with which to attack the problems encountered in the nation's 
schools. The investment of what is now billions of dollars in RDD&E 
activities — through Title III of the ESEA legislation, through the Co- 
operative Research Act, through the decade of support for the National 
Center for Educational Research and Development, the Bureau of Research, 
and the like — must be seen as evidence concerning the faith of the Con- 
gress and others in the power of the activities about which this paper 
is written. But the perpetual disappointment in the payoff of such 
activities, in the little good that can be traced directly to them, 
must be having its effects upon the strength of that commitment. Because 
the investment in the empirical investigation of RDD&E would be relatively 
small compared to the overall budget for such activities within the context 
of education , because the time span needed for such research is relatively 
short, and because the potential payoff is so high, extended work upon 
the access road and its support systems should be undertaken. Just as 
the microscope and the telescope and the laser beam and the cloud chamber 
have opened vast new arenas of research and development in other fields, 
with all of their attendant consequences for the social good, the develop- 
ment of the methodology of RDD&E could have similar consequences. 

The keepers of the roads are, after all, the ones responsible for 
the quality of the road system that exists. If good roads are wanted 
or needed, then good technology must be established for their creation. 

The empirical investigation of the domain of educational RDD&E would 
develop such technology . 
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GLOSSARY OF TERMS 

(ordered according to their appearance In the paper) 



Terms for Conceptual Set 1: Definitions 

OUTPUT. An Identifiable planned outcome of targeted work 

activities, with targeted work activities referring 
to actions directed toward the realization of projected 
goal states. 

EDUCATIONAL RDD&E. A coordinated set of problem-solving strate- 
gies designed to produce recognizable outputs that can 
be judged as to their quality and their contribution to 
the solution of educational problems. 

RESEARCH. A problem-solving strategy designed to produce 

reliable knowledge, that Is, facts, principles, theories, 
and laws that are generallzable and that can stand the 
test of empirical verification. 

RESEARCH: CONSTRUCT-ASCENDENT. Research which focuses upon the 

identification or clarification of constructs, their para- 
meters, or their Interrelationships. 

RESEARCH: TREATMENT-ASCENDENT. Research which focuses upon the 

Identification or clarification of treatment conditions, 
their parameters, or their effects upon targeted outcomes. 

EVALUATION, A problem-solving strategy designed to produce trus- 

worthy Information to be used In situation-specific decision 
making, that Is, observations, reports, and other outputs 
of formal or Informal measures that are collected with 
attention to the control of unwanted variance and organized 
so as to optimize their utility for a particular set of 
decision makers who have a particular decision or set of 
decisions to make within a particular time frame. 

EVALUATION: PROCESS. Evaluative efforts which focus upon the means 

by which a thing Is done. 

EVALUATION: PRODUCT. Evaluative efforts which focus upon the conse- 

quences of that which Is done. 

DEVELOPMENT. A problem-solving strategy designed to produce reliable 
technology, that is, procedures, materials, hardware, and 
organizational frameworks that have a known degree of success 
in bringing about a particular outcome or In performing a 
defined operation. 

DEVELOPMENT: REPRESENTA'^IONAL. Developmental efforts which focus 

upon the production of abstract or symbolic outputs, for 
example, theories, models , conceptual frameworks, literature 
reviews . 
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DEVELOPMENT: OPERATIONAL. Developmental efforts which focus 

upon the production of concrete or operational outputs, 
for example, instructional systems, management systems, 
teaching machines, computers. 

DIFFUSION. A problem-solving strategy designed to bring about 
the utilization of generalizable knowledge, reliable 
technology, or trustworthy information. As used here 
diffusion incorporates both the concepts of dissemina- 
tion and adoption. 

DIFFUSION: INTERPRETATIVE. Diffusion efforts which focus upon 

the utilization of knowledge and/or information. 

DIFFUSION: ADOPTIVE, Diffusion efforts which focus upon the 

utilization of technology. 



Terms fo:* Conceptual Set 2: Interactions 

RDD&E AS AN INTERDEPENDENT SET OF PROBLEM-SOLVING ACTIVITIES. A 

situation-specific mix of research, development, diffusion, 
and/or evaluation activities that are coordinated over 
time so as to yield solution to a particular educational 
problem* 

RDD&E AS AN INTRADEPENDENT SET OF PROBLEM-SOLVING ACTIVITIES. A 

situation-specific mix of research, development, diffusion, 
and/or evaluation activities that are coordinated over 
time so as to yield solutions to the particular problems 
encountered within a particular R,D,D, or E project* 

RDD&E AS A NESTED SET OF PROBLEM-SOLVING ACTIVITIES. A view of 

educational RDD&E as a set of problem-solving activities 
that can be brought to bear collectively on either the 
solution of large-scale educational problems, applied 
individually to small-scale problems, or applied collec- 
tively in service of any particular research, development, 
diffusion, or evaluation problem* 



Terms for Conceptual Set 3: Mechanisms 

MECHANISM* An organizational vehicle that carries RDD&E activities, 
that is, an institution, a program, a project* 

PROJECT. A formal or recognized unit of operation within which 
resources are brought to bear oa and activities coordi- 
nated in the solution of a given probletm within a given 
time period. 

PROJECT IDENTITY. A project dimension charact?rized by the pro- 
ject’s focus, setting, and linkage to the setting in which 
it rests. 



PROJECT COMPLEXITY. A project dimension characterized by the 
project’s resource base, its output commitments, and 
its related inter-program and/or inter-institutional 
alignments , 

PROGRAM. A formal or recognized unit of operation within which 
resources are brought to bear on and activities coordi- 
nated in the pursuit of two or more projects that have 
a common focus . 

PROGRAM: PROJECT-DEPENDENT. A program having its resource 

base tied to the specific projects that comprise it. 

PROGRAM: INSTITUTIONALIZED. A program havxng its resource base 

tied to a programmatic or goal-oriented effort, rather 
than to the specific projects which constitute it. 

RDD&E DEDICATED INSTITUTION. A formally constitiitcd and recog- 
nized unit of operation within which two or more programs 
of R,D,D, and/or E rest, and which assumes its primary 
identity from those programs. 



Terms for Conceptual Set 4: Outputs 

OUTPUT. An identif iablei planned outcome of targeted work 

activities, with targeted work activities referri^ng 
to actions directed toward the realization of projected 
goal states. 

PRODUCT. A tangible or "hard" outcome of work effort that 

survives in a form that makes it transportable, for 
example, an instructional system, a conceptual framework, 
a final report of a research or evaluation project. 

EVENT. An outcome of work effort that results in the occurrence 
of an observable transaction, for example,* a conference, 
a speech or seminar, consulting or counseling activities. 

CONDITION. An outcome of work effort that results in the creation 
of a desired circumstance, for example, a change of 
attitude on the part of students or their parents, 
mastery of information, an opportunity for teachers to 
participate in policy decisions. 

OUTPUT INDEX. The listing of outputs that emerge from a particular 
project. 

OUTPUT: FOCAL. An outcome of work effort expected by contractual 

obligation to emerge from a project. 

OUTPUT: TERMINAL. An outcome of work effrrt that constlt. tes an 

esflentlal part of or prerequlnite to a focal output. 



OUTPUT: COI-IPOI^ENT. An outcome of work effort that constitutes 

an element of or one step in the achievement of a terminal 
output . 

OUTPUT: FACILITATING. An outcome of work effort that is suppor-- 

tive to the generation of focal, terminal, or component 
outputs, but is not in itself an instance of such outputs. 

OUTPUT MAP. A graphic portrayal of the hierarchical and functional 
dependencies of outputs within a given project. 

OUTPUT STANDARDS. The criteria of excellence expected of ^*>r 
applied to a particular output. 



Terms for Conceptual Set 5: Operations 

FUNCTION: The largest meaningful classification of operations 

involved in the production of outputs. 

FUNCTION: POLICY SETTING. The establishment of standards for the 

focal output(s) of a project, the allocation of resources 
toward the realization of that output, and the establish- 
ment of guidelines as to the desired procedures to be 
followed in and anticipated consequences to derive from 
the project as a whole. 

FUNCTION: MANAGEMENT. The orchestration of the energy, time, 

materials, facilities, and information available to a 
project so as to optimize desired outcomes and minimize 
undesired outcomes. 

FUNCTION: PRODUCTION. The execution of the operations required to 

achieve a planned outcome of a project, given the con- 
straints and enablers provided by policy and management. 

ACTIVITY: The largest meaningful unit of work engaged in to carry 

out a particular function. 

TASK: The largest meaningful unit of work engaged in to carry 

out an activity. 

ACTION. The largest meaningful unit of work engaged in to carry 
out a task. 



Terms for Conceptual Set 6; Inputs 

ENABLER: Knowledge, skills, and sensitivities prerequisite to the 

performance of the operations required to produce a par- 
ticular output. 
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Terms for Conceptual S<it 7: Organization 

PROJECT ORGANIZATION: The manner in which su :h factors as 

structures , f unc tions , roles , responsibilities » j obs , 
etc, arc defined and positioned within a project in 
order to perform the work required to produc'i targeted 
outputs. 
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A CRITIQUE OF THE PAPER 



Glen Heathers 

Learning Researcli and Development Center 
University of Pittsburgh 



An important task in the study and Improvement c f edur:ational R&D 
is the development of a conceptual scheme for ordering data in this dif- 
fuse and relatively unformed field of activity. Schalock ana Sell are 
to be applauded for undertaking this formidable task and the framework 
they present, admittedly tentative and Incomplete, is a valuabT ^ con- 
tribution. The authors note that the framework already has gone through 
several revisions in the course of analysis of 20 representative R&D pro- 
jects. Further revisions and extensions clearly are in order and doubt- 
lessly will result from future work by its authors, and by others. Per- 
haps the chief function of this critique is to Identify limitations of 
the present framework, then to suggest ways in which it can be strength- 
ened . 



A difficulty in writing this critique is that the other Volumes in the 
series of reports on the Teaching Research project are not in hand. Those 
volumes present the case-study methodology and the body o" case data de- 
rived from study of the 20 selected projects. As such, they constitute 
the first test of the usefulness of the conceptual framework. The re- 
marks that follow should be viewed with this limitation in mind. 

The Schalock-Sell paper, following its introduction, documents the 
existing terminological muddle in educational RDD&E. That section does 
not attempt to resolve the muddle; its chief contribution is a valuable 
bibliography that readers can use in exploring problems of definition 
as represented in the literature. 

Central to their conceptual framework are the authors ^ definitions 
of R, D, D, and E. \^lle each definition begins with the phrase, "a 
problem solving strategy. . .,” it is the Intended outputs that anchor 
the definition. In this reviewer's judgment, this is a major limitation 
of the paper as a whole. Surely “adequate analysis of R&D projects re- 
quires specifying the features or steps in the problem solving process 
that constitute the entering phase in each definition. Case studies then 
could document how this process was carried out. Inasmuch as Schalock 
and Sell do not deal with this part of each definition, they leave a 
major analytic task still to be performed. Numerous articles in the 
literature offer analyses of problem solving task flows in educational 
R&D. An example is Hemphill's article on Educational Development (Urban 
Review, 1969, 23-27). It would have been useful had Schalock and 

Sell expanded their definitions at least by referring to such articles . 

The authors make a good case for the interactions among R, D, D, 
and E. The first D in the acronym, l.e., development, is a good case 
in point. Narrowly defined, educational development usually refers 
to building programs, procedures, or products involved in instruction. 



262 



More broadly defined, development applies also to building anything new 
in education, whether research methodologies and study designs, evalua- 
tion procedures and instruments, or diffusion methods and programs, 
l^ile the paper recognizes this confusion, a more specific treatment 
of it would have been helpful. 

The term diffusion, as the authors point out, has been use 1 to mean 
various things. Their definition, it should be noted, covers the uti- 
lization of knowledge or technology. The present writer leans toward 
the Cuba and Clark distinction between diffusion as covering dissemina- 
tion and demonstration, and adoption as covering trial, installation, 
and institutionalization, A difficulty seen in t.ieir formulation, as 
well, s that botl^ diffusion and adoption are product-oriented rather 
than consumer-urjented , There appears to be need for another term in 
the RDD^E acronym that provides for the design and conduct of local ed- 
ucational change programs that begin with identified needs of school 
systems, proceed to the analysis and evaluation of existing resources 
available nationally to meet those needs, then adapt and implement se- 
lected changes suited to the local situation. Perhaps the term uti- 
lization is as good a name for this process as can be found. What it 
is meant to stress, in contrast to the term diffusion, is a need focus 
rather than a product focus, 

Schalock and Sell offer a valuable illustration of the interaction 
of R, D, D, and E in the instance of a school system setting out to 
strengthen its reading program. The focus of their illustration is 
utilization as just outlined. Certainly it would be desirable for school 
systems to be able to carry out the specific research and development 
activities involved in building a good reading program. Presently, few 
school systems have the resources in time, money, and R&D competency to 
do this. The illustration, then, poses the critical question: to what 

extent can school systems be expected to perform major research and de- 
velopment activities? 

Certainly school systems, to mount significant change programs, 
must be concerned with designing, conducting, evaluating, and diffusing 
those programs. To accomplish these things, usually they require con- 
siderable expert help from outside sources. Fundamental work in the 
research and development r* ^cesses required to build new educational 
products such as reading -ricula usually is well beyond present re- 
sources existing in any ouc the largest school systems. An adequate con- 
ceptual framework for educational RDD&E would need to provide for ana- 
lyzing the conditions under which an institution, whether a school system 
or an'^ther educational agency, can successfully undertake given R&D ac- 
tivities, particularly in projects requiring major innovations falling 
within the domains of research and development. It is to be noted in 
this connection that the Schalock-Sell scheme is only sketchily built 
in two areas that bear directly on this matter: Conceptual Set 6. Inputs , 

and Conceptual Set 7j_ Organization . 

Conceptual Sets 3, 4, and 5 dealing with mechanisms, outputs, and 
operations appear to this reviewer to employ sets of terms that are, in 
the main, good at a general descriptive level. The distinctions between 
projects, programs, and institutions appear useful. Likewise the criteria 
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for project identity and project complexity seem justified as a first 
crude cut. It is more difficult to see from inspection the resolving 
power of the distinctions among product, event, and circumstance; or 
of those among focal, terminal, component, and facilitating outputs. 
Setting up a hieriirrny of functions, activities, tasks, and actions as 
classes of operations seems quite reasonable. With all of these cate- 
gorizations, the first test s how useful they are in case-study de- 
scription and analysis. The critical test, to come later, is the value 
of the distinctions when work proceeds to the analysis of factors dis- 
tinguishing more productive from less productive R&D activities. 

The output indexes (Tables 6-9) and the output maps (Figures 4 and 
do not bear identifiable relations with the category scheme indicated 
above except that Table 7 distinguishes products and events, while Tables 
8 and 9 distinguish policy setting, management, and production functions. 
The question that must be raised is how the analysis proceeds from the 
form of the output (usually a report) to its substance , How is the cat- 
egory system employed in the case analyses to yield such distinctions? 

An important limitation of the conceptual framework as presented 
concerns output standards. This limitation is fully recognized in the 
paper, with reasons given. From the information given about project 
interviews, it is evident that a major thrust of work on the analysis 
of educational RDD&E projects is needed in this critical area, Schalock 
and Sell can probably make an important contribution to clarifying the 
bases for such work by providing a detailed report on their efforts to 
obtain data on output standards. 

In their concluding section on **The Parable of the Roads,” Schalock 
and Sell come to grips with the central purposes of the Oregon Studies — 
to provide conceptual and data bases for strengthening educational RDD&E 
through research on R&D and through the training of R&D personnel. Ac- 
cepting their parable, they make a good case for their three roads, and 
for sustained research-and-development programs to Improve all three. 
Their paper is a significant step in that direction. Viewed in relation 
to the other reports of the Oregon Studies, it can be offered as an im- 
portant road-mapping job. 
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RESPONSE TO THE CRITIQUE 



H, Del Schalock and G, Rop,er Sell 



Heathers* critique is cript, sensitive, and pointed. In fewei 
pages than given to the Introduction to the paper he manages to com- 
ment not only on all facets of the paper, but to point up its essen- 
tial strengths and weaknesses as well. Because of the brevity of his 
critique, however, some of the points he makes are open to misunder- 
standing or misinterpretation. This applies to his criticisms more 
than to his plaudits, so our response will be simply to reiterate his 
criticisms as we understand them, and then to comment briefly. 

As we read Heathers* criticisms we find three kinds; (a^) omissions 
in the framework; (^) weaknesses in framework content; and (jc) weaknesses 
in content treatment. We will respond accordingly. 



Omissions 

Heathers sees two major omissions in the framework: (^) failure 

to take account, in our output-linked definitions of RDD&E, of *’, , , 
the features or steps in the problem solving process that constitute 
the entering phase in each definition** (p. 262); and (y failure to 
**, , , provide for analyzing the conditions under which an institution, 
whether a school system or another educational agency, can successfully 
undertake given R&D activities, particularly in projects requiring major 
innovations falling within the domains of research and development** 

(p, 263), 

Our response to the first charge of omission is **Maybe we* re guilty 
and maybe not. It depends on what Heathers means,** If he means that 
the framework makes no provision for the description or analys* of how 
st^ff within a project produce the outputs for which the project is 
responsible, which is the solution of problems in production, manage- 
ment, and policy setting, he is in error. The conceptual set dealing 
with operations (Conceptual Set 5) is designed precisely for that pur- 
pose , 



If he means that the framework makes no provision for the descrip- 
tion or analysis of all that goes into a project before it starts , that 
is, in the selection of the problem to be studied, in the selection of 
the broad strategies to be employed in its solution, etc,, he is cor- 
rect, The framework covers only ongoing project/program operations. 

Since it is likely that Heathers is making the latter point, his point 
is well taken, for in problem selection and in broad strategy for problem 
solution, long term gains in educational improvement are likely to be 
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To Lhu second charge of omission our response is '*It is an intended 
omission, because the ability to determine whether an institution or 
agency has the capability to carry out a specific R, D, D, or E activ- 
ity seems to us to be an analysis of a totally different kind.** This 
is not to say that such capability is unimportant. Tt is, especially 
if RDD&E are to become integral, ongoing parts of the educational enter- 
prise. But recognizing this importance is a different matter from say- 
ing that it should be included as a part of a framework designed to 
describe ongoing RDD&E activities. 



Weaknesses 



Heathers points to three weaknesses in the framework: (£) the 

conceptualization of diffusion (p. 263); (y the treatment of inputs 
to and the organization of projects and programs (p. 263); and 
(O the classification scheme for outputs (p. 264). 

Our response to the first charge is one of confusion, in regard 
both to what Heathers says and what we think he means. He confuses 
us in what he says by shifting between the position we have taken and 
the position that Cuba and Clark have taken in relation to diffusion. 

This is especially so in the sentence **A difficulty seen in their form- 
ulation, as well, is that both diffusion and adoption are product oriented.*’ 
Is this in reference to our position (if so it is not correct) or is it 
in reference to the position of Cuba and Clark? 

More important is our confusion over what we think Heathers means. 

We think he thinks that we think differently than he thinks (that should 
add clarity to the discussioni) . Heathers implies that our view is coun- 
ter to the distinction between dissemination and adoption proposed by 
Cuba and Clark. This is not so. We include both under the term diffusion, 
but restrict their usage to the diffusion of different kinds of R, D, & E 
outputs (see pp. 217, 222 for first order definitions, and Table 5, p. 222 
for suggested second and third order definitions) . The rationale for 
the position we have taken appears on p. 209. Given the uncertain- 
ties surrounding Heathers’ comments it Is hard to respond to his pro- 
posal to coin the term ’’utilization** to cover the . . design and 



IVJhile the conceptual framework was developed so as to provide 
descriptions of ongoing RDD&E projects/programs, the methodology that 
carries the application of the framework can be applied as readily to 
the planning stage of a project as it can to its operational srage. 

Given the nature of the methodology (see Vol. V in the series reporting 
the Oregon Studies on RDD&E) all that is needed Is to specify the ex- 
pected outputs from the planning effort, for example, the identification 
of a problem or the completion of a proposal, and then apply the system- 
atic inquiry strategies that are incorporated within the methodology to 
elicit the ’’problem solving” steps followed in arriving at the definition 
of the problem or the development of the program. 
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conduct of local educational change programs/’ 

In response to Heathers’ comment about the short treatment given 
project inputs and organization in the framework we simply acknowledge 
it. The purpose of including Conceptual Sets 6 and 7 at this point in 
time was simply to ’’draw attention" to the input and organizational 
dimensions of project and program operation for . . little has been 
done thus far in the development of the framework . . ."in those areas 
(see p. 244). 

Our response to Heathers’ view that "It is more difficult to see from 
inspection the revolving power of the distinctions among product, event, 
and circumstance; or of those among focal, terminal, component, and 
facilitating outputs" (p. 264) must be a bit more complex. The labels 
given these category sets may not appear to be particularly powerful, 
or the categories may seem to be at too fine a detail to be functional, 
or they may not "square" with how outputs are typically thought of. But 
they do in fact represent category sets that evolved against the realities 
of the field and were found to be functional in handling those realities. 
Space does not permit a detailed discussion of their origins or use, but 
for a sense of the distinctions being made between kinds of outputs (pro- 
ducts, events, conditions) the reader should review the definitions that 
appear in the GLOSSARY that accompanies the paper. The rationale under- 
lying the four level category set used to describe the positioning or 
patterning of the outputs found within any project can be found on 
p. 233 of the paper. As Heathers rigutly points out, however, even 
though the category sets have proved useful in the description of on- 
going projects, their critical test will come when the data from the 
study have been analyzed and when "... v’ork proceeds to the analysis 
of factors distinguishing more productive from less productive R&D ac- 
tivities . " 



Weak Treatments 



In Heathers’ judgment there are also three areas in the paper that 
received relatively weak treatment, even though the dimensions of the 
framework being described at those junctures apparently are adequate: 

(^) the discussion of the implications of our definition of development 
(p. 262); (Jb) the treatment of standards held for outputs (p. 264), 
and (O the illustrations provided of output indexes and maps (p. 264). 

In our opinion his judgment is accurate on the first two counts, but 
only partly so on the last. 

In the first case, development was dealt with only at the level of 
definition (see pp. 216 and 222 of the paper). In the second case, 
due to difficulties encountered in obtaining sufficient data base for 
category development-refinement, category sets simply had not been de- 
veloped at the time of writing (see pp. 237 and 238 of the paper). 

In the third case, however. Heathers is both accurate and inaccurate 
in his comments. He is accurate when he points to the lack of identifiable 
correspondence between the categories used to refer to the patterning of 
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outputs in the body of the paper (focal, terminal, component, facili- 
tntinpj and their appearance in the output maps (Tables A cui S) jsed 
to illustrate the patterning effect. This was intended to be sho^^m in 
the tables by block coding: however, it did not appear in the drat? copy 
reviewed by Heathers, and we have used editorial license to correct this 
omission in the final version printed here. 

So much for the heart of Heathers’ critique. Comment upon a rel- 
atively peripberal observation will close our response to it. Heathers 
observed (p, 262 ) that the chief contribution of the review of litera- 
ture offered within the paper is its value as a bibliography. With un- 
abashed acclaim, we view it a bit differently. It is not in any way an 
exhaustive review, and thus is not a particularly good bibliography. 

What it aimed to do and what we think it does is to highlight the de- 
finitional issues that currently exist in the field — a task which many 
such reviews somehow fail to do. 
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SUMMARY CRITIQUE 



CRITIQUER: Francis S. Chase 

Educational Consultant, 
Professor (and Dean) Emeritus 
The University of Chicago 







POSTSCRIPT TO TIO* R,D^I),^K PAIM-US 



Francis S. Chase 
Educational Consultant , 
Professor (and Dean) Emeritus 
University of Chicago 



It seems unlikely that any except the most conscientious and per- 
sistent readers will have, waded through the four long papers, the sev- 
eral critiques, and the responses to the critiques to reach this point. 

Those who have done so should be spared this postscript. The selective 
samplers, who have come through with a hop, some skips, and an occasional 
jump, probably have gathered what is most worth gleaning. One paper only 
requires concentrated attention to garner what it offers; and from another, 
fruitful generalizations present themselves readily; beyond these the 
harvest is not abundant. This memorandum may serve to reassure the skim- 
mers — and occasionally lead them back for a second glance; but it is ad- 
dressed chiefly to those eccentrics who leap to tlie end for clues useful 
in backtracking. 

In spite of the long association of the writers through several 
conferences and opportunities for revision and rerevision (by authors 
of at least three of the papers) following reactions from presentations 
to two selected audiences, the papers do not fit together in any system- 
atic way; and only two of them — those by Gideonse and the team of Schalock 
anc Sell — appear to have been improved significantly since the early drafts. 

The \>jary reader will do well to start with Gideonse 's sprightly 
philosophical and commonse.isical commentary and proceed from there through 
the tightly organized and painstakingly constructed framework by Schalock 
and Sell. Where the wary reader goes from here may be left to his own 
discretion; but it may prove illuminating to sample the critiques before 
the other two papers. There is a good chance that a number of readers 
will find the reviewers’ comments at least as rewarding as the papers 
which evoked the critiques. (This assumption is based on faith in the 
powers of the critiquers as the critiques have not been seen by this 
commentator . ) 

It is unlikely that anything can be said in this postscript that 
has not already been said at least as pointedly in one or more of the 
papers and critiques; bu' , ^n deference to the assignment, an attempt 
must be made to assess the contributions of the papers, severally and 
collectively. Three related ruestions may be posed as points of entry: 

1. Do the papers, individually and collectively, add anything 
of importance to an understanding of what and how research 
and development concepts and operations may contribute to 
the improvement of education? 

2. Will the reader who attends conscientiously to the content 
of the four papers come away with a clearer preception of 
the essential nature of research , development , diffusion , 
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and evaluation in education and of how they relate to each 
other and to the planning and management of educational 
operations? 

3. Do the several treatments complement or reinforce each other 
so that taken together they afford a consistent and coherent 
view of the four processes as elements essential to any 
systematic attempt to increase educational effectiveness? 

Brief answers to these questions , in reverse order , may be stated 
as follows: 

Interpaper coherence appears slight or nonexistent; positions in 
one paper are reinforced seldom by another paper; and such complemen- 
tarity as exists seems to spring more from chance than fron» design. 

Discerning readers, pitrhaps, will get a clearer fix ihan they pre- 
viously had on the four proc^ ssec , and their interrelationships , from 
the definitions, differentiations, and similarit.es as expounded by 
Glass and Worthen in their rather pedantic, but occasionally clarifying, 
attempts to set these processes in a general framework of educational 
inquiry; and, from close attention to Schalock and Sell, they may ac- 
quire sets of concepts useful for examining the process€)S from varying 
perspectives and begin to perceive the makings of a framework for empir- 
ical investigation of educational research and development. Gideonse, 
arguing not from definition of terms but from a consum'ir-oriented view- 
point, offers provocative ideas for educational R&D policy-making and 
planning; and Briggs tries, without notable success, to point out prac- 
tical ways of implementing development and diffusion in education. 

Contributions to an understanding of what r^-isearch and development 
(interpreted broadly to include the four processes) can contribute to 
educational improvement, and how th(^ potential contributions can best 
be realized, are skimpier tnan might have been expected from the pooled 
efforts over a period of months by authors so well qualified by ability 
and experience. Some inferences may be drawn from Briggs* inadequately 
developed concept of consortium-based diffusior and deyelopment, and a 
greater number from Gideonse *s incisive comments on policy for uoaca^ 
tional RuD. 

The foregoing statements admittedly are no more than abbreviated 
personal judgments which are set forth without evidence, but which the 
reader may readily confirm, negate, or modify on the basis of his own 
reactions to the papers^. Ic may be informative to move from these 
summary answers to questions arbitrarily posed by this commentator to 
an attempt at a rough analysis and assessment of what the papers actually 
contain. The major content of the papers may be classified either under 
the heading of definitions and conceptual frameworks or under propositions 
bearing on R&D policy, planning, management, or personnel training • 

Of the more than 250 pages of typescript in the four papers, over 
40 a. is devoted to attempts to define and place In ediiratlonal confexts 
the four processes of rescartli, development, dlffufMon, and evaluatlf)n. 
This treatment Is limited largely to two papers: tlwit i^y Glass arul 
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Wr>r!lirn v/IjIcI) <.»x/jrnl n('S rosoiirch, development, d I f ftift i on , and eviiluation 
within ilte field of educational inquiry In order to Illuminate 

the relationships of these processes to each other and to other inquiry- 
related activities; anu that of Schalock and Sell, which presents the 
four processes as "a carefully defined set of concepts and an equally 
carefully defined network cf relationships between concepts.” The other 
two ])."'ner.-^ address themselves to definitions only as necessary to the 
<attainment of other purposes. 

The quality of these treatments, judged in their own terms, is 
relatively high in comparison with previous attempts to define the 
meanings and applications of the four terms in the field of education. 
Glass and Ivorthen do a scholarly, If somewhat tedious, job of moving 
from definitions centered on the purposes ana activities involved to a 
series of attempts to relate the processes to each other and to the gen- 
eral domain of inquiry in education. Along the way they take time out 
to reaffirm or question views expressed by other writirs with regard to 
the appropriate methodology for one or more of the activities or with 
regard to the application of the outcomes. It is a game beloved by 
academicians, played for the edification of other academicians and for 
".he kudos bestowed by the academic community. One may question some of 
their conclusions, detect small errors here and there, or insist on his 
ov;n allegedly superior version; but one probably will concede that the 
game la played according to the rules favored in communities of educa- 
tional researchers, \7itI1 results that compare favorably v/ith most such 
exercises . 

Schalock and Sell adopt a Ifferent approach and produce signifi- 
cantly different results. They set their discussion against the back- 
ground of the actual and potential contributions of research, develop- 
ment, diffusion, and evaluation to the improvement of education and 
move through a review of the literature to the major thrust of the paper, 
which is to provide a framework for viewing educational RDD&E. Defini- 
tions of the four terms are constructed with care, and with an eye to 
their subsequent use. The first level definitions are clear and succinct, 
and provide useful characterizations of the four terms. The second level 
definitions and illustrative third order definitions carry the analysis 
further, and into more debatable ground. 

They treat definitions as the first of seven conceptual sets rela*:ed 
to the outputs that the four processes are intended to produce. Their 
second conceptucl set is denoted "interactions” and offers provocative 
concepts of RDD&E as interdependent, intradepandent , and nested sets of 
problem-solving activities. Figure 1 provides an illuminating schematic 
illustration of a network 01 these problem solving activities in support 
of a research project. From this point on, the authors stretch out their 
set of categories until they become a little thin; but, nevertheless, 
come up with some interesting observations. The treatment of the mech- 
anisms or vehicles that carry RDD&E in education is clarifying, but the 
figures seem poorly adapted to picturing the dynamic interactions in- 
v^'lved. The discussion with regard to classes of outputs is intriguing 
aiU may have some practical applications. The other categories likewise 
provide interesting and potentially useful ways of analyzing or examining 
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research and development operate on?. 

The utility of these essays into definition is not entirely clear. 

I can see some usefulness to students who are encountering for the first 
time the concepts of these processes in educational research and develop- 
ment. The approach of Schalock and Sell may rJso produce some assist- 
ance in analysis and evaluation of actual operations. In summary, it 
may be said that: The definitional essays are more than ample in quantity, 

almost to the po^at of superfluity; the quality is comparatively high 
in comparison with other attempts to get at the meanings and applications 
of the four terms; and the utixity is somewhat dubious with respect to 
the Gl^ss-Worthen paper and potentially high for the application of the 
Schal jck-Sell framework to empirical studies and assc' ants of educa- 
tional research and development operations. 

The preceding commentary has centered largely on definitions of the 
four processes to which the major part of two papers is devoted and to 
the resultant contributions to the understanding of, and communication 
about, research and development in education. 

The discussion of strategics for control and implementation of 
research and development processes in education are found mainly in the 
papers by Gideonse and Briggs. The former adopts a more or less phil- 
osophical approach from which he spins off a whole series of generali- 
zations. The effect may be illustrated by stringing together some key 
generalizations which are scattered throughout the paper, thus: 

... Science as it is practiced in managing the support of 
education is as much a ^ )cial and political activity as ... 
a scientific one ..., [therefore] ... the conditions causing 
practitioners and policy-makers to attend to the ideas emerging 
from science must be established parallel to and as part of 
the support of science itself ..• [Ic follows that] educational 
research and development must be conceived in terms of the 
consumers and clients it is supposed to serve. • . . [wh-* ch 
means that] *•. . xamination of the conditions, reasons, and 
requirements for change within the educational system Itself 
is the first necessary step to the improvement process ... 

[and in decision-making] ... status hierarchies ... are 
replaced by transactional processes of confrontation , bar- 
gaining, or negotiation among equals. 

These views are not as novel as Gideonse seems to :hink, but they 
are stated with unusual force and aptness, and so related that impli- 
cations become apparent. Briggs and Gideonse reinforce each other at 
several points, although Gideonse's paper is more philosophical-poli- 
tical, and Briggs' is beamed more to the practical, even though it does 
not constitute an adequate guide to action. Briggs and Gideonse ag-ee 
on such points as that R&D planning operations should represent a trans- 
action among equals , and that the delivery system Is essential to effec- 
tive development. The formulations of both Gideonse and Briggs can be 
accommodated within the Schalock-Sell framework. 



In the first section of his paper, Brlj^r,s develops the hypotliesls 
tliat specif Ic changes in education depend largely upon who wants the 
cliangi’* the general methods used for producing tlic change, and what re- 
st)ur<‘es for (diange are employed. He uses six categories of change ini- 
tiators from which, for some curious reason, he onlts local and state 
edijcatlcm authorities, and professional organizations. With the aid 
of a table, he sets forth the methods, resources, and mechanisms which 
are operative or applicable to each category of initiators of change. 

While the discussion is thought-provoking, it ignores some important 
approaches to change which have had influence in the past, such as the 
pioneering and laboratory schools, the contributions made by organiza- 
tior s su:h as cooperative school councils (a kind of consortium), and 
the impact of movements and events such as the Child Development Move- 
ment in the early part of the century, the Great Depression of the 1930'a, 
the Progress! /e Education Movement, and the Civil Rights Movement. 

When we examine the contributions to the application or implementa- 
tion of the four processes in education, we find some Interesting con- 
trasts, Glass and V/orthen offer proposals for implementation which bear 
no discernible relationship to the main body of their paper. Their pro- 
posals boil do\^m to four points : 

1. Increase expenditures for basic educational research 
and the training of researchers in the universities, 
and involve all of the social sciences. 

2. Move schooling *'out of the grants economy and under the 
influence of the market mechanism'* (which the authors 
claim will have such an effect on educational practice 
as to make insignificant all other approaches). 

3. Development and diffusion of marketable products should 
be conducted primarily in the private sector; but uni- 
versities and regional laboratories possess much talent 
for developing curriculum materials. 

4. Educational evaluation does not fit into the reward 
structure within educational research and, therefore, 
may require the establishment of new types of organi- 
zations- -presumably nonprofit. 

In short, the solution offered by Glass and Worthen for educational 
renewal is simply: to move schooling along with applied educational 
research, development, and diffusion into the private sector; to support 
generously basic educational research and some curriculum development, 
chiefly in the universities; and to hope that specialized organizations 
will arise to provide evaluation. Having thus declared themselves, they 
acknowledge some problems in applying the market mechanism to education 
(due, in large part, apparently, to corruption Introduced by "the ed* 
ucators’ desire to eliminate competition"); but they brusti these pro- 
blems lightly aside in their contemplation of the beauty of education 
constantly transformed by the benign operations of free enterprise 




i^nllghtened by university contributions to basic knowledge. Their 
conclusions are not well supported either by evidence or nrj^iinent and 
appear too often as dogmatic assertions. This is notably true of the 
statement alluded to in item 2 above (of which more later). 

The contributions of Schalock and Sell to implementation consist 
largely of the argument that educational improvement requires continued 
support for educational research, careful attention to the building of 
capabilities for systematic research and development, and development 
of the necessary support systems for that purpose. One of the support 
services suggested is that of a personnel training center. 

Gideonse suggests three implications of what he calls the ’’market 
mechanism”, namely: (£) the cultivation of research techniques associ- 

ated with the identification and definition of operational need; (y a 
deliberate attempt to stimulate the creation of autonomous Institutions, 
scattered across the country; and (y continual searching out of alter- 
native ways of cultivating research capabilities. He also offers sug- 
gestions for decision structure, based on the idea that research and 
development in education is a social and political process as well as 
a scientific one. Among the more plausible ideas are the holding of 
public hearings across the country on a regular basis to enable citizens 
to present their ideas about what is needed, the use of ’’administrative 
lobbyists” as a way of assuring continuous contact with Important in- 
terest groups in education, and much closer and more continual Congres- 
sional oversight of educational research and development. Gideonse 
concludes his well-organized, consistent, and persuasively presented 
view of educational research and development with a vision of some other 
means toward the goal of building an educational system based on cur- 
rent knowledge, with an operating philosophy shifted from systems main- 
tenance to systems renewal and increasingly accountable to multiple clients, 
both individual and societal. He then offers some provocative comments 
on governance, assessment, and accountability, the development of re- 
search capabilities, the creation of adaptive organizations, linkage 
mechanisms, and manpower development. He summarizes his own contri- 
butions succinctly by saying ’’the analysis addresses the importance of 
identifying and defining client needs first, the availability of delivery 
systems to serve those needs second, and only then what innovations or 
knowledge might be required to create something that could be delivered 
to fill client needs.” 

Briggs’ suggestions for implementation of development and diffusion 
are offered chiefly in connection with his proposed ’’consortium-based 
model for development and diffusion.” He sets forth a number of basic 
assumptions upon which the proposed model rests, including an increas- 
ing press for major changes in education that will lead to the forming 
of consortia to bring .ibout the agreed upon changes, heavy involvement 
of the lay public in decision making, and the choice of goals which will 
require massive new development and evaluation efforts and new working 
relationships between education and industry. He then moves to what he 
calls "steps of the model” which go into such details as translating 
goals into objectives, identifying assumed entering competencies, pre- 
paring posttests, selecting the media, and so on through 17 items, 
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endin^» v»tth the training of teachers and the permanent reassessing of 
needs, lie concliuics with what he calls ’’some elenents for success of 
tlie model/’ but for some obscure reason does not address himself forth- 
rightly to the enormous problems encountered in putting together an ef- 
fertive (consortium composed of several kinds of organizations. 

The big problems in the building of effective consortia of the scope 
implied Inhere in the long processes of planning and negotiations required 
(a) to bring about broad agreement on the needs to be met, the direction 
of change, and the goals of action; (y to assess the capabilities and 
primary functions of the several agencies as a basis for differentiation 
of functions, and the playing of complementary roles for the accomplish- 
ment of tlie agreed upon purposes; (c) to assure equitable contributions 
by the several agencies to a pool of substantial resources, with each 
agency's contribution being determined by consideration both of its re- 
sources and of the returns which it may anticipate on its investment in 
the common effort; (^) to obtain agreement on the establishment of a 
decision-making body, or bodies, which will give all members equal op- 
portunity to participate in policy making — and at the same time facili- 
tate decisions based on analyzed information and shared values; (e) to 
work out means through which each participating agency will be assured 
of benefits in terms of Improved performance of its own functions, ap- 
proval by its own constituency and sources of support, and improved morale 
of its staff. These are conditions not to be achieved easily or without 
long and painful negotiations and the exercise of resourceful and imagin- 
ative leadership. Moreover, these conditions are difficult to attain at 
all under funding policies which reward competition among agencies and 
foster rivalry in respect to credit for accomplishments. 

It is doubtful that any critiquer will offer judgments as devastat- 
ing as those incorporated in the papers by Gideonse and the team of Glass 
and Worthen. Jideonse, early in his treatise, expresses doubt that ed- 
ucational R&D is likely to be advanced by **worry and concern” about the 
state of the art or ”the nature and interrelationships of R&D functions 
and processes.” Taken literally that view renders futile a large part 
of the labors by Glass and Worthen, Schalock and Sell. In a footnote, 
Gideonse attributes the excessive concern (pecullai. to education) with 
R&D functions to the ”long prevalent methodological consciousness of ed- 
ucational research and the basic insecurity about our pretensions to 
scientific status.” One cannot help wondering whether the current fashion 
of "modeling,” to which Gideonse and Briggs succumb, does not spring from 
the same source. It is difficult to see what is added to Gideonse 's 
strong presentation by mislabeling it a '^market model.” 

Glass and Worthen in their most emphasized generalization declaim 
as follows; 

T he net effect on educational practice of all the scholarly 
papers and conferences on educational change which could be 
produced, all of the laws Congress could_pass, and all of the 
federal programs which could be mounted would be insignificant 
beside the impetus ^or chan;^e that would result fi.om moving 
s choolir^ out of the grants ec onomy and under the influence 
of the market mechanism. 
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) h i .s taken at face value dlMposes of tin* wholi? set of 

pu(H‘rs ami (ritl(|iies <aml of all ^’overnnent-.supporLed research and develop- 
"v-fif ,i.s v,a* 1 I . riie I'.enora I I /.a 1 1 on , however. Is pul forv/ard more as an 
artlilo of faith than ac a hypothesis f(»r Inqviirv, evidential analysis, 

(^r tirc.unent . Tlu* arvTura*nt , svu'h as it is, reaches the lowest level in 
liir siati’neni that: "1 ho knee-jerk bureaucratic response of passing a 

law or rn(unuinp a p.overnpieii t program whenever a pndilem Is encountered 
is wasteftil and a potential fallur’ as a neans of pronotlnp edvirational 
cliaiiju’ • '* The authors follow this with another assertion tliat in their 
opinion ’V'ovlru’, school inp, ovit of tlie grants economy into a market economy 
would ca\ise edvica t i oiia I development and diffusion to flourish.” They build 
tlieir (*ase for niovinp, schooling; Into a *Vi/irket e('onony” and for obtaining 
edu(*ational research and development throiu!h f)rlv/itc Industry throup,h 
ove rl app Itir, , thoup.li not identlc'al arp,unents. Fhe major points seem to 
he as follows: (a) bureaucracies (in whicli eclvic at tonal Institutions nay 

be included) are noloriouHly ineffective In produclne or nurturing change; 
(b) most applied research and devch^pnent In oilier fields is done by pri- 
vate? indvistry; (c) tiic liard lines facing t. lie grant economy argue for re- 
ducing the strain on public support for education tlirniigti the transfer 
of responslb 1 1 1 ty for applied researcli and development to Industry; (d) 
there are a number of trends toward putting education in the private 
sector . 

Glass and Wortlieui appear to overlook tlie fact that it cook a long 
struggle, growing out of tlie inadequacies of private schooling, to 
create the public school systems in the United f»tates and elsewhere. 

To be sure, the "vovicher” system was not then In vogue, but there were 
Charley scliools. They similarly overlook or lightly brush aside the 
unimpressive educational research and development record made by industry 
when it has moved from tlie "liardware” into the ^software” of educational 
development . 

It is legitimate to ask how the papers might have been st re ngt honed . 
The first suggestion that occurs to the weary reader is simply ”by sl.ort- 
ening.” This night have been accomplished by condensing and consolidat- 
ing the review >f the literature into a single paper and combining or 
reducing the number of definitions and the number of conceptual categories. 

On the other hand, some further elaboration would have been useful 
for such salient proposals as det ennlning the educa t ional needs of con- 
sumers, perfecting delivery systems, and establishing consortia for 
development and diffusion. Further attention might also have been given 
to the establishment of networks of communication and mutual stimulation 
among researchers, developers, evaluators, diffusers, and practitioners 
(or intermediate consumers). A section on tlie mix of capabilities re- 
quired for effective applied research and development in education would 
have been a helpful springboard to implications for training of personnel 
for educational R&D. 
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A GUIDE TO THE OREGON STUDIES IN EDUCATIONAL RDD&E 



Volume I 

SUMMARY REPORT 

An introduction to and overview of the Oregon Studies as a 
whole. The volume contains an outline of the history of the 
Studies, the rationale around which they were designed, the 
context within which they were carried out, and the 
prootdures followed in their execution. It also co.itains a 
description of the projects selected for study, the rationale 
underlying their selection, the criteria and procedures used 
in their selection, and an overview o( the data collected on 
each project. Finally, the volume contains an introduction 
to the “case profllcs" that house the data collected on each 
project, the results of all cross-project analyses, and the 
summary recomnwndations that have been made relative to 
training and the continued study of educational RDD&E 
activities. A brief description of the case study method- 
ology developed within the Studies, an overview of a 
process whereby investigators may query computer-stored 
data flies and original interview statements to obtain 
information bearing upon specific questions relating to 
training, manpower, policy, and work performance, and 
supporting data accompany the volume. 



Volume II 

THE UTERATURE OF EDUCATIONAL RDD&E 

A compendium of existing literature that defines, describes, 
differentiates, or relates the activities labeled educational 
research, development, diffusion, evaluation, and various 
combinations thereof. The articles within the volume are 
introduced as a collection. Linking passages provide an 
interpretive context both for individual articles and for the 
sets into which they have been grouped. 



Volume III 

CONCEPTUAL FRAMEWORKS FOR VIEWING 
EDUCATIONAL RDD&E 

A collection of papers which provide the conceptual 
uj.derpinnings to the Oregon Studies. It contains three 
papers commissioned by the Studies as a basis for con- 
ceptual development, and a paper by staff from Teaching 
Research that describes the conceptual frame that guided 
and grew with the empirical thrust of the Studies. Each of 



the papers is a major document which defines, differ- 
entiates, and relates one or more facets of educational 
RDD&E and provides a supporting rationale for the 
position adopted. Each p«pcr is accompanied by a formal 
critique, and the set u' papers is accompanied by ao 
introductory and summary critique. 



Volume IV 

PROFILES OF EXEMPLARY PROJECTS 
IN EDUCATIONAL RDD&E 

A collection' of twenty case profiles that form the data 
base in the Oregon Studies. Printed in three parts, the 
profiles describe five research projects, seven development 
projects, thrbe evaluation projects, and five diffusion 
projects. Each profile contains descriptions of the structure 
and function of the project being analyzed, the specific 
outputs expected to emerge from it, the operations 
required to produce each output, and the knowledges, 
skills, and sensitivities judged to be essential to the 
performance of those operations. In addition, each profile 
contains sections dealing with the “dynamics" of project 
operations and implications that derive from the project for 
preservice staff training. The projects described range from 
small, two-man efforts within university settings to very 
large school district “projects" employing several dozen 
staff members. Eighteen of the twenty projects described 
were judged to be illustrative of the kinds of RDD&E 
activities likely to occur within the context of education in 
the future. The twenty projects account for analyses 
around 298 project outputs and interviews with 134 
professional staff members. 



Volume V 

A METHODOLOGY FOR THE STUDY 
OF EDUCATIONAL RDD&E 

A detailed description of the most refined form of the data 
collection methodology developed within the Studies, 
directions to guide its use, and the decision rules needed for 
the volume to function as a users manual. The volume 
includes information on procedures used in site contact, 
site preparation, data reduction and analysis, and profile 
preparation. It also includes information on the category 
sets used in data reduction and the computerized data files 
that contain or provide access to all data collected in the 
Studies. 



Copies of any or all of these volumes 
may be retained at cost from 
Teaching Research 
Monmouth, Oregon 97361 
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